ESGF_User_FAQ
No, this is just the FAQ section; a guide to the most common problems and questions. You will find the complete guide here .
Please note that the CCSM4 and CESM1 models are distributed from the NCAR site, which at this time works differently than all others sites: you will have to log onto that site (www.earthsystemgrid.org) with your openid once, after which you should be able to run the wget scripts.
The OpenDAP urls are available on a per file bases. Once search results (datasets) have been added to your Data Cart, on that pane use the "Show Files" link for a dataset, to reveal the files and OpenDAP links for each.
download the data ?
The new ESGF system does not require the user to reply to any confirmation email. You can immediately follow the process for downloading the data as described in the ESGF Data Download Guide .
We encourage all users that are currently registered with the "pcmdi3" server to start using a new "pcmdi9" openid, as the pcmdi3 server has been retired. If your username was say, "foobar"... on pcmdi3, you should have been assigned a new ESGF openid in the form: https://pcmdi9.llnl.gov/esgf-idp/openid/foobar. The best way to check is to login to the pcmdi9 node (http://pcmdi9.llnl.gov/esgf-web-fe/login) and put in your new ESGF openid, as described above, and you should then be able to enter your password and complete the login. If not, please create an account for yourself (https://pcmdi9.llnl.gov/esgf-web-fe/createAccount) and use those credentials where credentials are required.
Please note that it might be necessary to re-register the new "pcmdi9" openid for CMIP5 access - please see instructions .
The registration workflow doesn't work very well on Windows machines, even when using a recent version of FireFox . You may find that after being redirected to the registration page, clicking on the registration button causes the following error:
HTTP Status 500 - java.lang.Exception: HTTP POST request failed: url= https://pcmdi9.llnl.gov/esgf-idp/secure/registrationService.htm error=HTTP/1.1 500 Internal Server Error
Possible solutions:
-
Quit FireFox , empty its cache, try again
-
Restart FireFox, open a "New Private Window" (FireFox does not remember its cache and cookies in this mode)
-
Use Internet Explorer on Windows
-
Log onto the pcmdi9 portal, go the Account page, and apply for CMIP5 Research group there
This could be either a problem with the user account, or with the server setup where the user registered:
- Possibly, the user data (openid, first name, last name, etc.) contain non-standard characters, which will make the login fail. The user should change the data him/herself, or contact the server administrator
- Or, the server is not setup correctly. The administrator should check for these possible problems:
- Certificate expired (host certificate or CA certificate)
- Root CA not in the truststore or not in the federation certificate repository
- Server OpenID provider URL not in the whitelist of OpenID relying party
- Server clock sync issue
- Firewall side effects
- The file esg-trustore.ts does NOT match the file jssecacerts in the JAVA installation directory
Make sure that the checkbox "Search all Sites" is checked.
In general, search results should not depend on the ESGF portal but in reality there are sometimes differences. Our node administrators and developers are continuously working on improvements but running a worldwide data grid isn't easy. The numbers of datasets "seen" by each ESGF portal are listed in the ESGF Data Visibility API for the most important projects. There, look for part "indexes" since ESGF portals are "index nodes". The information in the ESGF Data Visibility API is refreshed every 5 hours.
selected variables ?
Please note: the procedure used to generate a wget script that only contains files for a specified variable has changed with the latest ESGF release. Not all ESGF portals have upgraded to the latest version of the software yet. If in doubt, please use the pcmdi9.llnl.gov or esg-datanode.jpl.nasa.gov portals.
Please follow these steps:
- Use any of the facet constraints on the left to return a set of matching datasets (each dataset containing many files, possibly for more than one variable).
- Add the datasets you are interested in, one at a time, to your Data Cart
- Click on "Data Cart" to view your datasets, and "Show Files" to show the files in each dataset.
- If the datasets contain more files than you need (for other variables), you can use the text box on top to sub-select the files by a matching string. For example, enter "tas" to only sub-select those files that contain "tas" in their filename. Another option is to restrict to a specific variable by entering for example "variable:tas". In either case, you must click the "Search" button for the constraint to take effect.
- Click on the radio button "Filter Over Text".
- Finally, click on each dataset "WGET" link, or on the global "WGET All Selected" link, to generate a wget script containing only those files.
This denotes lack of proper authorization, either because the user is not properly authorized for CMIP5 Research access, or because the data on the server is not configured to allow access to CMIP5 Research users.
-
Example error message:
Connecting to esg-datanode.jpl.nasa.gov|137.78.210.36|:443... connected.HTTP request sent, awaiting response... 403 Forbidden 2012-02-03 04:52:26 ERROR 403: Forbidden.
-
Solution: please perform an http download of a single file from the web interface before attempting to run a wget script, as described in the Registration and Download Guide . This operation will guide you through the process of enrolling with the appropriate data access group. (There is a more robust and streamline group registration process under development, soon to be released.)
The wget script does not trust the certificate of the server.
- Solution: the script needs to be run with the -i option
- certificate unknown : signals that the server does not trust the certificate issued by the MyProxy CA
* Example error message:
Connecting to test-datanode.jpl.nasa.gov|137.78.210.101|:443... connected.
OpenSSL: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
Unable to establish SSL connection.
download failed
* Solution: you might need to re-install the set of trusted ESGF certificates:
* remove everything in the directory ~/.esg/*
* run the wget script to download a new set of trusted certificates
* Solution: contact the [ ESGF users mailing ](mailto:esgf-user@lists.llnl.gov) list to notify the ESGF administrators there might be a problem with the server certificate
In a less common scenario it is possible that you might not have access to the VeriSign CA and therefore you cannot download the required files from rainbow.
To turn off the security checking create (or edit) the file $HOME/.wgetrc
and add this line to the bottom of it: ( _ please note this is a security
problem, since you will be sending your data to untrusted servers! Ask your
Sysadmin about this procedure, you could have problems depending on your
institutions security regulations _ ):
check_certificate = off
This causes wget to send your _ personal _ ESGF certificate to pretty much anyone who asks for it.
The md5 checksum does not match
-
Example error message:
md5 failed!re-downloading...Downloading
-
Solution: use -p to _ preserve _ the file even if the md5 checksum doesn't match. Some institutions are known to alter the files after publication without updating their metadata, e.g. the checksum. In those cases it's impossible to know if the file was downloaded correctly. The _ -p _ flag does check the md5 and provides the result of that comparison, though it leaves the file as it was downloaded. We can't do anything else, you'll have to contact the author and ask for the checksum or blindly _ trust _ what you've downloaded.
Failure unspecified at GSS-API level [Caused by: Unknown CA]]]
- Solution: try to remove everything under ~/.esg and run the MyProxy or wget script command again.
failed [Caused by: org.ietf.jgss.GSSException, major code: 11, minor code: 0
-
Example error message:
The certificate expires in less than 8 hour(s). Renewing...Please give your OpenID (hit ENTER to accept default) [https://myserver/example/username]? https://pcmdi9.llnl.gov/esgf-idp/openid/....... MyProxy Password? Retrieving Credentials...MyProxy get failed. [Caused by: Authentication failed [Caused by: org.ietf.jgss.GSSException, major code: 11, minor code: 0 major string: General failure, unspecified at GSSAPI level minor string: None [Caused by: Bad certificate (java.security.SignatureException: SHA-1/RSA/PKCS#1: Not initialized)]]] Use --help to display help. Certificate could not be retrieved
-
Solution: make sure that you have installed Oracle Java 1.7 or above, try running "java -version" and "java -showversion".
through wget script or MyProxyLogon
-
Example error message:
Retrieving Credentials...MyProxy get failed. [Caused by: connect timed out] Use --help to display help. Certificate could not be retrieved -
Solution: please make sure that your wget script can connect to the ESGF MyProxy server, for example on host "ipcc-ar5.dkrz.de" and port "7512" do the following:
echo | telnet ipcc-ar5.dkrz.de 7512-
This is the output expected if you can connect successfully:
Trying 136.172.30.10...Connected to bd110.dkrz.de (136.172.30.10). Escape character is '^]'. Connection closed by foreign host.
-
And this is the output expected if you cannot make the connection:
Trying 136.172.30.10...
telnet: connect to address 136.172.30.10: Connection refused
telnet: Unable to connect to remote host: Connection refused
If this is the case, you need to contact with your system administrator and tell him you need to access ipcc-ar5.dkrz.de over port 7512 (TCP) (or the server and port you are trying to connect to). (see node port matrix )
-
Example error message:
No ESG Credentials found in /Users/daniele/.esg/credentials.pemOpenSSL 1.0.1c 10 May 2012 WARNING: ESGF Host certificate checking might not be compatible with OpenSSL 1.0+ Retrieving Federation Certificates...mv: /Users/daniele/.esg/esg_trusted_certificates: No such file or directory done! Please give your OpenID (hit ENTER to accept default) [https://myserver/example/username]? https://pcmdi9.llnl.gov/esgf-idp/openid/..... MyProxy Password? Retrieving Credentials...Invalid or corrupt jarfile /Users/..../.esg/getcert.jar Certificate could not be retrieved
-
Solution 1: remove all contents of the directory ~/.esg and try running the wget script again
-
Solution 2: the wget application installed on your system might not be compiled with SSL. You can check this by issuing "wget -help" and investigating wether or not there are some SSL options. If not, talk to your system administrator about installing a new version of wget with SSL support
-
Solution 3: your system might not be trusting VeriSign, the Certificate Authority that issued the certificate of "rainbow". Try to do the following:
- Create a file ~/.wgetrc with the following content: check_certificate = off . This line is instructing wget to accept all certificates, from any host.
- Remove all contents of the directory ~/.esg
- Run the wget script again
-
Solution 4: your system may be behind a proxy that does not allow connections to the rainbow server. Try setting the following environment variables and run the wget script again:
- export http_proxy="my-proxy.dmz:80"
- export https_proxy="my-proxy.dmz:443"
-
Solution 5: you can try to obtain a certificate on another system (for example, clicking on the link " MyProxy, and then copy the certificate from the other system onto the old system where you want to run, in the location ~/.esg/credentials.pem
- This error is most commonly seen on MacOS systems, and is caused by wget not finding a standard CA certificate list or bundle. If wget was installed from MacPorts , the easiest solution to this is to install the 'curl-ca-bundle' package, and then edit either /usr/local/etc/wgetrc or ~/.wgetrc to contain the following line:
CA_CERTIFICATE=/opt/local/share/curl/curl-ca-bundle.crt
- If a wget script that is working fine on a non-MacOS system is consistently failing with a 401 error on a MacOS system, the following things should be checked
- wget is the latest version: this problem was seen on wget 1.13, but not on a system with wget 1.14, on a MacOS 10.7 system
- GnuTLS v2 is in use, NOT GnuTLS 3: GnuTLS 3 has problems dealing with a large number of CA certificates
- If all else fails, try upgrading from MacOS 10.7 to MacOS 10.8
RSA/ECB/PKCS1Padding is not available from provider Cryptix"
- The problem is that the wget script needs Oracle Java 1.7 or newer to work. It wont work with OpenJRE where you'll get cryptographic provider errors. Try running "java -showversion" to check your Java.
- The problem is again that the wget script needs Oracle Java 1.7 or newer to work. Because of security reasons one of the SSL protocol versions had to be disabled on ESGF servers. If your Java cannot switch to another protocol version, a handshake is not possible. Try running "java -showversion" to check your Java.
