M*D17A3 product - download problems

[Nowosad]

I'm unable to download M*D17A3 product (layer: Gridded 1 Km Annual Net Productivity). I've tried ftp option:

[Thu Feb 23 14:41:01 2017] Retrieving list of files from NASA server
Trying to reach ftp server - attempt 1

...and http option:

[Thu Feb 23 14:41:38 2017] Retrieving list of files from NASA server
Trying to reach http server - attempt 1

While both of messages claim that server is down, however I can access https://e4ftl01.cr.usgs.gov/ without any problems. I also was able to download a different product - the problem occurs only with M*D17A3.

Thanks for your work on this package and let me know if you need some more information.

[lbusett]

Hi, thanks for signaling this. We'll look into it ASAP.

Can you tell us when did you last update the package ?
Also, can you please visit your earthdata "profile" page (https://urs.earthdata.nasa.gov/profile), click on "My Applications" and ensure that "LP DAAC Data Pool" is authorized ? If not, click on "Approve More Applications, search for "LP DAAC Data Pool" in the list and approve it (If you already did it, forget about this).

Lorenzo

[Nowosad]

Hi @lbusett,

1. I've installed the github version of the package today.
2. "LP DAAC Data Pool" is authorized on "My Applications" webpage.

[lbusett]

ok, thanks. I'll see what's happening tomorrow and come back to you,

[lbusett]

ok, thanks. I'll see what's happening tomorrow and come back to you.

[lbusett]

Oh, well. I probably already solved it - easier than expected.
MOD17A3 has version 005.5 and the urls were looking for version 005 .

can you try installing the recently updated devel repo:

install_github("lbusett/MODIStsp", ref = "devel")

and see if now it works ?

[Nowosad]

I'll let you know tomorrow. Thanks for your work!

[Nowosad]

@lbusett I've installed a new version (devtools::install_github("lbusett/MODIStsp", ref='devel')) and tried to download MOD17A3 data for the entire possible dates and extent. Data is available on the server (https://e4ftl01.cr.usgs.gov/MOLT/MOD17A3.055/), but when I tried to download it using MODIStsp, I get messages like:
No available data for year: 2002 for Sensor Terra in selected dates. (for Terra) or Trying to reach http server - attempt 41 (for Aqua)

[lbusett]

Hi,

for the "Terra" problem: Is it only 2002 not working, or all years ?

For AQUA, instead, It seems that v005 is no longer availble (or not available right now: see https://e4ftl01.cr.usgs.gov/MOLA/).

Do you have any particular reason for not using v006 (Products M*17A3H) ? At least it seems that the same version available for the two platforms.

Lorenzo

[Nowosad]

@lbusett, problem occurs for all years. My settings (I've only removed my user name and password):

I cannot use v006 using MODIStsp - it is impossible to change a version when I use "Ecosystem Variables - Net Primary Productivity" as a category. "5" is the only choice.

[lbusett]

Hi,

sorry: I didn't explain myself very well....

In the "Ecosystem variables" category, you'll find two products: MOD17A3 and MOD17A3H. MOD17A3H is v006 of MOD17A3 (I think), though thewy changed the resolution to 500 m. Don't know why they changed the name....

https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/mod17a3h_v006

Lorenzo

PS: I see the same problem on MOD17A3: I'm looking into it.

[Nowosad]

I have the same problem on MOD17A3H.
I've also tried to download files for one year by myself and next use "Download server: offline", but it's not working as well.

[lbusett]

Oks.

I probably found the problem. I'll tell you when you can re-install.

[lbusett]

Ok. Can you try:

install_github("lbusett/MODIStsp", ref = "hotfix/MOD17A3")

(on http download - ftp has still some problems)

[hgiche]

Thanks Lorenzo - by installing the above version with the hotfix, I was able to download the missing first layer for the first date for M*D13Q1 NDVI v6 data.

[lbusett]

Should be fixed on commit 8f45814 - both on devel and master. Error was due to check on dates using "<" instead than "<=". Therefore, images with date 20XX-01-01 weren't recognized

You can install again the devel or master branches: they were both fixed.

Feel free to reopen if you still have problems.

[Nowosad]

@lbusett I've performed simple test - MOD17A3H data between 2000-01-01 and 2002-01-01 for h=10 and v=05:

Error in download.file(url = remote_filename, destfile = local_filename,  : 
  cannot download all files
In addition: Warning messages:
1: In download.file(url = remote_filename, destfile = local_filename,  :
  downloaded length 0 != reported length 9975613
2: In download.file(url = remote_filename, destfile = local_filename,  :
  URL 'ftp://ladsweb.nascom.nasa.gov/allData/6/MOD17A3H/2000/001/MOD17A3H.A2000001.h10v05.006.2015141012954.hdf': status was 'Failure when receiving data from the peer'
[Mon Feb 27 11:09:15 2017] Download Error - Retrying...
[Mon Feb 27 11:09:25 2017] Downloading Terra Files for date 2000_01_01 : 1 of 1
trying URL 'ftp://ladsweb.nascom.nasa.gov/allData/6/MOD17A3H/2000/001/MOD17A3H.A2000001.h10v05.006.2015141012954.hdf'
Content type 'unknown' length 9975613 bytes (9.5 MB)

downloaded 0 bytes

[lbusett]

Http download is working for me. Did you try also that ?
I'll check tomorrow what's happening on ftp, though.

By the way: Can you tell me if you are usng "aria2c" acceleration ?

Lorenzo

[lbusett]

Hi Nowosad,

I checked also on ftp but I wasn't able to replicate the issue: I was able to download the data without problems. Can you retry, either by ftp or http ?

Maybe you just hit a temporary downtime of the ftp server OR you have a partially downloaded file in your PC that is messing things up (that should be accounted for, but nonetheless I'd suggest you to try to delete everything from your current output folders)

Let us know,

Lorenzo

[Nowosad]

Hi @lbusett - will check tomorrow and let you know.

[Nowosad]

@lbusett I was able to download hdf's using http for two sample tests - MOD17A3H (ver. 6) and MOD17A3 (ver. 55) data between 2000-01-01 and 2002-01-01 for h=10 and v=05.

In the same time, I've found another thing - Spatial Extent in the program is limitted to be between 1 and 35 for horizontal and 0 to 17 for vertical. However, the files on the site https://e4ftl01.cr.usgs.gov/MOLT/MOD17A3.055/2000.01.01/ starts with horizontal = 00.

[lbusett]

Ok. Glad that it works now.

Thanks also for reporting the tiles issue. Seems that no one tried to download tile 0 so far. Will fix that ASAP.

[Nowosad]

Thanks

[lbusett]

Tiles bug corrected on commit b00e8c9

[hgiche]

With the hotfix version download speed was reduced dramatically and I got the error
Warning messages:
1: running command '"aria2c"' had status 1

I have just downloaded the "devel" version and I am still getting the same warning message
1: running command '"aria2c"' had status 1

[lbusett]

that's strange. I opened a new issue on this: #65

[lbusett]

To clarify: Are you seeing the download speed drop also on devel, or only the aria2c message ?

[hgiche]

The download speed has also dropped in the devel version.

[lbusett]

Hi Helene,
I updated this issue: #65 Please use that to communicate on this topic.
Lorenzo