Deb install downloads fail with NoSectionError: No section: 'esgf_credential'

[basilveerman]

I have installed synda following the deb instructions and updated my credentials in /etc/synda/sdt/credentials.conf.

Running

synda get cmip5.output1.MPI-M.MPI-ESM-LR.historical.day.atmos.day.r3i1p1.tasmax.v20111006

Results in:

Traceback (most recent call last):
  File "/usr/bin/synda", line 162, in <module>
    status=sdtiaction.actions[args.subcommand](args)
  File "/usr/share/python/synda/sdt/bin/sdtiaction.py", line 119, in get
    import sdlogon, sdrfile, sddeferredafter, sddirectdownload, syndautils, humanize, sdconfig, os, sdconst
  File "/usr/share/python/synda/sdt/bin/sdlogon.py", line 122, in <module>
    openid=sdconfig.config.get('esgf_credential','openid')
  File "/usr/lib/python2.7/ConfigParser.py", line 607, in get
    raise NoSectionError(section)
NoSectionError: No section: 'esgf_credential'

Are additional configuration steps needed?

[ghost]

You need to run the command with sudo

e.g.

sudo synda get pmip3.output.MPI-M.MPI-ESM-P.lgm.monClim.atmos.Aclim.r1i1p2.v20140428

Also because of servers downtime, you may need to change indexes in sdt.conf file

vi /etc/synda/sdt/sdt.conf

#indexes=esgf-data.dkrz.de
#default_index=esgf-data.dkrz.de

indexes=esgf-index1.ceda.ac.uk
default_index=esgf-index1.ceda.ac.uk

[ghost]

You can run command below to display an explicit error message instead of Python Exception

sudo sed -i -e "s/install/get','install/" /usr/share/python/synda/sdt/bin/sdconst.py 

I added the fix in the doc below
https://github.com/Prodiguer/synda/blob/master/sdt/doc/deb_install.md#patch

Regarding the fact that you need sudo to run 'synda get' command,
I opened the issue below:

#20

It should be possible to use 'synda get' with normal user if we store ESGF credential for each user
(currently, only one ESGF credential is stored in /etc/synda/sdt)

[basilveerman]

That's great, thanks. Using sudo resolves the issue, but I would agree that superuser privileges should not be required to initiate a download.

I do have another issue now however...

When starting a test download, I am now seeing:

synda get cmip5.output1.MPI-M.MPI-ESM-MR.historical.day.atmos.day.r3i1p1.v20120503 tasmax   
5 file(s) will be downloaded for a total size of 1.2 GB
[...]
Transfer failed with error 1 (did you subscribe to the required role/group ? (e.g. cmip5_research, cordex_research))
Download failed (http://aims3.llnl.gov/thredds/fileServer/cmip5_css02_data/cmip5/output1/MPI-M/MPI-ESM-MR/historical/day/atmos/day/r3i1p1/tasmax/1/tasmax_day_MPI-ESM-MR_historical_r3i1p1_19900101-19991231.nc)

When using a browser to download that dataset, I am redirected to log in, then when that completes the download starts. I have the role "CMIP5 Research"

Are there other conditions which could cause this error?

[ghost]

I successfully downloaded the files using my openid, so I guess something must differ between my certificate and your certificate.

Can you print the output of the command below ?
(it to display roles informations associated with your certificate)

sudo openssl x509 -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_issuer,no_pubkey,no_sigdump,no_aux -text -noout -in /var/tmp/synda/sdt/.esg/credentials.pem

If the certificate is missing, you can run this command to create it
$ sudo synda certificate renew

[basilveerman]

It seems that it took a while for the role to propagate. I had just applied it a couple hours before attempting this but if I try now I do not receive an error.

To avoid using sudo I actually ended up re-installing from source on a new Ubutnu 16.04 machine with python 2.7.11.

Upon completing that, a synda certificate renew would result in this error:

'sdlogon.sh' script returned an error

status=4
stdout=/home/bveerman/sdt/bin/myproxyclient logon -T -s pcmdi.llnl.gov -p 7512 -l bveerman -o /home/bveerman/.esg/credentials.pem -b
2016-05-31 16:52:32 - ERR009 - Error occured while retrieving certificate
stderr=Traceback (most recent call last):
  File "/home/bveerman/sdt/bin/myproxyclient", line 11, in <module>
    sys.exit(main())
  File "/home/bveerman/sdt/local/lib/python2.7/site-packages/myproxy/script.py", line 196, in main
    do_logon(myproxy, options)      
  File "/home/bveerman/sdt/local/lib/python2.7/site-packages/myproxy/script.py", line 220, in do_logon
    updateTrustRoots=options.trustroots)
  File "/home/bveerman/sdt/local/lib/python2.7/site-packages/myproxy/client.py", line 1384, in logon
    **getTrustRootsKw)
  File "/home/bveerman/sdt/local/lib/python2.7/site-packages/myproxy/client.py", line 1533, in getTrustRoots
    conn = self._initConnection(verifyPeerWithTrustRoots=(not bootstrap))
  File "/home/bveerman/sdt/local/lib/python2.7/site-packages/myproxy/client.py", line 699, in _initConnection
    context = SSL.Context(SSL.SSLv3_METHOD)
  File "build/bdist.linux-x86_64/egg/OpenSSL/SSL.py", line 489, in __init__
  File "build/bdist.linux-x86_64/egg/OpenSSL/_util.py", line 48, in exception_from_error_queue
OpenSSL.SSL.Error: [('SSL routines', 'SSL_CTX_new', 'null ssl method passed')]

Error occurs while renewing certificate (code=SYDLOGON-001,message=Cannot retrieve certificate from ESGF (hostname=pcmdi.llnl.gov,port=7512))

I then manually downloaded the credentials with a system (deb) install of MyProxyClient and synda was able to use that credentials.pem and it worked.

I did notice that a source install of MyProxyClient would then fail to renew the credentials at version 1.3.1, but the upgrading the version to 1.4.3 solved the issues with MyProxyClient as well as synda certificate renew.

Thanks for your help!

[ghost]

It seems that it took a while for the role to propagate. I had just applied it a couple hours before attempting this but if I try now I do not receive an error.

Good to know

To avoid using sudo I actually ended up re-installing from source on a new Ubutnu 16.04 machine with python 2.7.11.

Well done, that's the way to go until I remove the sudo dependency in system packages

I did notice that a source install of MyProxyClient would then fail to renew the credentials at version 1.3.1, but the upgrading the version to 1.4.3 solved the issues with MyProxyClient as well as synda certificate renew.

Indeed, the myproxyclient version was frozen to the 1.3.1 for no reason.
I changed the installation script to use the latest instead.

Thanks for your feedback