Unable to download via NSIDC web site

[Gabriel-Desharnais]

Hello,

I wrote a Python2 script to download via MOLT source and everything is working just fine, but when I port this script to download from MOST data source (on NSIDC web site) I can't get it to work.
My pymodis version is 2.02 and my code is the folowing

from pymodis import downmodis
modisDown= downmodis.downModis(rep_download,MOTPASSE,UTILISATEUR,tiles='h12v04,h13v04',today='2016.01.01',delta=1,product='MOD10A2.006',url='https://n5eil01u.ecs.nsidc.org')
modisDown.connect()
print modisDown.getAllDays()

[Gabriel-Desharnais]

After taking a good look at it I see 2 problem:

1. the path was wrong I'm sorry after adding the keyword path="MOST/MOD10A2.006" the url accesed by the module is right
2. the authentication process aspens sooner on NSIDC server then on USGS.

[Gabriel-Desharnais]

Foud solution I'll be working on a fork next week

[lucadelu]

@Gabriel-Desharnais probably setting the path variable should fix your problem. At least for me is working.

from pymodis import downmodis
modisDown= downmodis.downModis('/tmp/', tiles='h12v04,h13v04',today='2016.01.01',delta=1,product='MOD10A2.006',url='https://n5eil01u.ecs.nsidc.org', path='MOST')
modisDown.connect()
modisDown.downloadsAllDay()

ls -lath /tmp/
total 742M
-rw-r--r--  1 lucadelu femgis  192 Jan 27 14:47 listfileMOD10A2.006.txt
-rw-r--r--  1 lucadelu femgis 6.4K Jan 27 14:47 MOD10A2.A2016001.h12v04.006.2016183134152.hdf.xml
-rw-r--r--  1 lucadelu femgis 704K Jan 27 14:47 MOD10A2.A2016001.h13v04.006.2016183134228.hdf
-rw-r--r--  1 lucadelu femgis 6.4K Jan 27 14:47 MOD10A2.A2016001.h13v04.006.2016183134228.hdf.xml
-rw-r--r--  1 lucadelu femgis 1.1M Jan 27 14:47 MOD10A2.A2016001.h12v04.006.2016183134152.hdf

[cfusting]

I'm also getting failure on connect. Same error from the command line and below called directly from Python.

>>> modisDown=downmodis.downModis('/users/c/f/cfusting/scratch/snow',user='cfusting',password='***',tiles='h20v02',today='2011.01.01',enddate='2014-12-31',delta=1,product='MOD10A2.006',url='https://n5eil01u.ecs.nsidc.org', path='MOST')
>>> modisDown.connect()
/gpfs1/home/c/f/cfusting/modis/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/gpfs1/home/c/f/cfusting/modis/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/gpfs1/home/c/f/cfusting/modis/lib/python2.7/site-packages/pymodis/downmodis.py", line 341, in connect
    raise Exception("There are some troubles with the server. "
Exception: There are some troubles with the server. The directory seems to be empty

[lucadelu]

@cfusting, @Gabriel-Desharnais

sorry I'm not able to reproduce your error.. could you try again? maybe some trouble with the server...

[cfusting]

Same response today. Here is the output from modisMOD10A2.006.log:

2017-02-14 08:31:45,205 - DEBUG - Starting new HTTPS connection (1): n5eil01u.ecs.nsidc.org 2017-02-14 08:31:45,412 - DEBUG - https://n5eil01u.ecs.nsidc.org:443 "GET /MOST/MOD10A2.006 HTTP/1.1" 302 431 2017-02-14 08:31:45,413 - DEBUG - Starting new HTTPS connection (1): urs.earthdata.nasa.gov 2017-02-14 08:31:45,595 - DEBUG - https://urs.earthdata.nasa.gov:443 "GET /oauth/authorize?client_id=_JLuwMHxb2xX6NwYTb4dRA&response_type=code&redirect_uri=https%3A%2F%2Fn5eil01u.ecs.nsidc.org%2FOPS%2Fredirect&state=aHR0cHM6Ly9uNWVpbDAxdS5lY3MubnNpZGMub3JnL01PU1QvTU9EMTBBMi4wMDY HTTP/1.1" 200 None

[Gabriel-Desharnais]

I made a module to bulk download images from NSICD website, I looked at doing a fork of pymodis but I needed a fast solution. Modis python suite. It's a bit dirty, (I will work on it) but at least it's working.

I really don't get how @lucadelu could successfully download from NSICD web site since they don't use basic authentication and that they ask you to login when listing the date available for a product. downmodis.py just fundamentally can't deal with that.

Greetings,

[timstevens1]

I am having similar issues. Called from the command line here:

modis_download.py -U 'tim_stevens' -P '****' -t h20v02 -u 'https://n5eil01u.ecs.nsidc.org' -p 'MOD10A2.006' -s 'MOST' -f 2011.01.01 -e 2014.12.31 -D 1 ../test_data/precip
WxPython missing, no GUI enabled
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):
  File "/usr/local/bin/modis_download.py", line 154, in <module>
    main()
  File "/usr/local/bin/modis_download.py", line 145, in main
    modisOgg.connect()
  File "/usr/local/lib/python2.7/dist-packages/pymodis/downmodis.py", line 341, in connect
    raise Exception("There are some troubles with the server. "
Exception: There are some troubles with the server. The directory seems to be empty

[lucadelu]

@timstevens1 I found an error on your call, you should modify the dates like this 2011-01-01, so your know command line should be like this

modis_download.py -U 'tim_stevens' -P '****' -t h20v02 -u 'https://n5eil01u.ecs.nsidc.org' -p 'MOD10A2.006' -s 'MOST' -f 2011-01-01 -e 2014-12-31 -D 1 ../test_data/precip

[lucadelu]

@Gabriel-Desharnais sorry for the daley in the answer, but did you try my last version from trunk?

[cljohn91]

I'm getting the same issues described above. Running bash on Mac OSX El Capitan V 10.11.6. I declared the path as described above.

modis_download.py -U 'cljohn91' -P 'ABCDE' -t h16v02,h15v02 -u 'https://n5eil01u.ecs.nsidc.org' -p 'MOD10A2.006' -s 'MOST' -f 2016-01-01 -e 2016-12-31 . WARNING: Python GDAL library not found, please install it to check data downloaded with pyModis qualitymodis and convertmodis_gdal modules not enabled, maybe Python GDAL is missing WxPython missing, no GUI enabled /usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) /usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) Traceback (most recent call last): File "/usr/local/bin/modis_download.py", line 154, in <module> main() File "/usr/local/bin/modis_download.py", line 145, in main modisOgg.connect() File "/usr/local/lib/python2.7/site-packages/pymodis/downmodis.py", line 341, in connect raise Exception("There are some troubles with the server. " Exception: There are some troubles with the server. The directory seems to be empty

[StefaniePes]

Hi,
I'm having the same troubles as described above.

modis_download.py -U 'myuser' -P '******' -t h18v04 -u 'https://n5eil01u.ecs.nsidc.org' -p 'MOD10A1.006' -s 'MOST' -f 2012-10-01 -e 2012-12-31 -D 1 .
Traceback (most recent call last):
File "/usr/local/bin/modis_download.py", line 153, in
main()
File "/usr/local/bin/modis_download.py", line 144, in main
modisOgg.connect()
File "/usr/local/lib/python2.7/dist-packages/pymodis/downmodis.py", line 380, in connect
raise Exception("There are some troubles with the server. "
Exception: There are some troubles with the server. The directory seems to be empty

Is there any solution to that? Thank you very much.

Best Stefanie

[lucadelu]

Hi @StefaniePes @cljohn91 I'm not able to get your same problem. Below some suggestions:

• please be sure to have enable the access to your NASA websites http://www.pymodis.org/info.html#requirements
• please try getting data from the browser to be sure that your password is correct and the application is enables
• please try using Python3, this should work better than in Python2.

Let me know

[StefaniePes]

Hi,
thanks for your quick response. I have access and I can download data from the browser. I have enabled the applications “LP DAAC Data Pool” and “Earthdata Search”. With Python3 I don't get any Exception but I don't receive any files either.

With the listfileMOD10A1.006.txt file empty the log-file says:
2018-01-11 16:45:11,006 - DEBUG - Starting new HTTPS connection (1): n5eil01u.ecs.nsidc.org
2018-01-11 16:45:11,617 - DEBUG - https://n5eil01u.ecs.nsidc.org:443 "GET /MOST/MOD10A1.006/2012.12.31 HTTP/1.1" 302 446
2018-01-11 16:45:11,620 - DEBUG - Starting new HTTPS connection (1): urs.earthdata.nasa.gov
2018-01-11 16:45:12,237 - DEBUG - https://urs.earthdata.nasa.gov:443 "GET /oauth/authorize?client_id=_JLuwMHxb2xX6NwYTb4dRA&response_type=code&redirect_uri=https%3A%2F%2Fn5eil01u.ecs.nsidc.org%2FOPS%2Fredirect&state=aHR0cHM6Ly9uNWVpbDAxdS5lY3MubnNpZGMub3JnL01PU1QvTU9EMTBBMS4wMDYvMjAxMi4xMi4zMQ HTTP/1.1" 200 None

Also tried from within python. No error but no files either.

from pymodis import downmodis
WxPython missing, no GUI enabled
modisDown= downmodis.downModis('tmp/', tiles='h18v04', today='2012.10.01', delta=1, product='MOD10A1.006', url='https://n5eil01u.ecs.nsidc.org', path='MOST', enddate='2012.10.03', user='', password='')
modisDown.connect()
modisDown.downloadsAllDay()
ls tmp/
listfileMOD10A1.006.txt modisMOD10A1.006.log

Thanks for your support!
Best Stefanie

PS: with 'https://e4ftl01.cr.usgs.gov' and MOD11A1 it works fine...

[lucadelu]

Hi @StefaniePes

I run your same command and it is working for me

-rw-r--r--  1 lucadelu femgis 7.6M Jan 11 17:58  MOD10A1.A2012366.h18v04.006.2016140153843.hdf
-rw-r--r--  1 lucadelu femgis 7.5K Jan 11 17:58  MOD10A1.A2012366.h18v04.006.2016140153843.hdf.xml
-rw-r--r--  1 lucadelu femgis 7.6M Jan 11 17:59  MOD10A1.A2012365.h18v04.006.2016140153644.hdf
-rw-r--r--  1 lucadelu femgis 7.5K Jan 11 17:59  MOD10A1.A2012365.h18v04.006.2016140153644.hdf.xml
-rw-r--r--  1 lucadelu femgis 7.6M Jan 11 17:59  MOD10A1.A2012364.h18v04.006.2016140152939.hdf
-rw-r--r--  1 lucadelu femgis 7.5K Jan 11 17:59  MOD10A1.A2012364.h18v04.006.2016140152939.hdf.xml
-rw-r--r--  1 lucadelu femgis  288 Jan 11 17:59  listfileMOD10A1.006.txt
-rw-r--r--  1 lucadelu femgis 8.3K Jan 11 17:59  modisMOD10A1.006.log

So there is something wrong in your configuration. Some other suggestions:

• are you behind a proxy?
• did you enable also NSIDC_DATAPOOL_OPS application? this is required for https://n5eil01u.ecs.nsidc.org

[StefaniePes]

NSIDC_DATAPOOL_OPS enabled, not behind a proxy. Tried from two machines and from the university network and from home. What might I be missing?

[cfusting]

@StefaniePes If you are having the same issue I did you might try the software developed by @Gabriel-Desharnais - did the trick for me.

[StefaniePes]

Thanks a lot! Works for me too.

[liaduarte]

Hi!

I am trying to use pymodis in command line and it gives me exactly the same server error described in this post. I confirm that I follow all the suggestions proposed and I have everything well configured (from the suggestions). So I am wondering why I cannot connecting to the server. Have you find come other way?

[lucadelu]

@liaduarte

I'm really sorry, because I'm not able to reproduce the error. I will try to add more debug in the next days.