SRTM1 not accessible from USGS anymore [Was: Make error on OSX]

[slarosa]

Testing the python package on my OSX and I am getting the following error:

make: Nothing to be done for download'.`

and

FAILURE: No input filenames specified.
make: *** [SRTM1.vrt] Error 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/larosa/dev/ext-libs/elevation/datasource.py", line 154, in clip
    datasource_root = seed(bounds=bounds, **kwargs)
  File "/Users/larosa/dev/ext-libs/elevation/datasource.py", line 137, in seed
    util.check_call_make(datasource_root, targets=['all'])
  File "/Users/larosa/dev/ext-libs/elevation/util.py", line 49, in wrapper
    return wrapped(path, *args, **kwargs)
  File "/Users/larosa/dev/ext-libs/elevation/util.py", line 80, in check_call_make
    subprocess.check_call(cmd, shell=True)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'make -C /Users/larosa/Library/Caches/elevation/SRTM1 all ' returned non-zero exit status 2

Although the package works very well on Linux.

[alexamici]

As a matter of fact MacOSX is my primary development platform and I use Elevation regularly on it, so it must be something specific to your setup. Can you please report the output of:

eio selfcheck

Furthermore what version are you using? The PyPi release or GitHub master?

[timstallmann]

I'm actually getting the exact same error on Mac OS X. eio was working previously for me as of a few weeks ago. selfcheck reports Your system is ready.. This is on eio version 0.9.9, installed via pip.

[slarosa]

@alexamici thanks for your quick reply.

I am using elevation module (master version) as an external library, as follow:

import sys
sys.path.append("/Users/larosa/dev/ext-libs")

import elevation
elevation.clip(bounds=(12.35, 41.8, 12.65, 42), output='/Users/larosa/Rome-DEM.tif')

and running selfcheck I get:

from elevation import cli
cli.selfcheck()
Your system is ready.

[alexamici]

Ok, I managed to reproduce the error on my machine after doing a eio distclean.

Looks like the data provider usgs.gov switched to authorised downloads only :( This isn't a MacOSX bug you should not be able to download new tiles from Linux as well.

@slarosa can you please confirm? Note that Elevation caches downloaded tiles, so please test clipping in a location that you never tried before.

[alexamici]

I'm going to make an emergency release to change the default product to SRTM3 and add a warning about SRTM1. Unfortunately the data policy was quite clear that direct access was not supported.

[slarosa]

@slarosa can you please confirm? Note that Elevation caches downloaded tiles, so please test clipping in a location that you never tried before.

:-( just after a distclean Linux fails to get SRTM1 as well.

[alexamici]

Looks like someone had the brilliant idea to download all of SRTMGL1 from the "direct" server and arguably USGS got upset: https://twitter.com/cartocalypse/status/759855534379929600

Getting Elevation to work with USGS credentials might be possible, but quite some work :(

[slarosa]

Hi @alexamici, testing against master version, I again get the error. I see the *.tif file in cache directory but it has 0 byte as size. Monitoring the spooldirectory I see that the *.zip file is downloaded correctly before cleaning.
I always get "Your system is ready." by running cli.selfcheck().

Any hint? Thank you!

[alexamici]

Release 0.9.10 and master both download SRTM3 by default that is expected to work. If you force product='SRTM1' you'll have the same error as before.

Anyway, HTTP errors leaves 0 byte files that stop Elevation from trying downloading the tile again, this is intended behaviour for non existing tiles, unfortunately also temporary HTTP errors trigger the same behaviour. Your best bet is to run eio distclean, otherwise you may remove the unwanted 0 bytes files yourself.

[slarosa]

@alexamici sorry for the noise it was my fault, the elevation package works fine with SRTM3 by default.

Looking forward to hear good from you for SRTM1....

Thank you.

[alexamici]

Looks like SRTMGL1 is now on S3! https://aws.amazon.com/public-data-sets/terrain/

[alexamici]

Integrated Amazon SRTM1 mirror and making the 30m the default again.

[slarosa]

Thank you @alexamici .

[falahfakhri-Iraq]

Hi, I'm getting the following error, Windows 11, py 3.9

Traceback (most recent call last):

File "C:\Users\FALAH FAKHRI\AppData\Local\Temp/ipykernel_21944/2656178845.py", line 1, in
elevation.clip(bounds=(12.35, 41.8, 12.65, 42), output=path + 'Rome-DEM.tif')

File "C:\Users\FALAH FAKHRI\AppData\Local\Programs\Python\Python39\lib\site-packages\elevation\datasource.py", line 216, in clip
datasource_root = seed(bounds=bounds, **kwargs)

File "C:\Users\FALAH FAKHRI\AppData\Local\Programs\Python\Python39\lib\site-packages\elevation\datasource.py", line 188, in seed
ensure_tiles(datasource_root, ensure_tiles_names, **kwargs)

File "C:\Users\FALAH FAKHRI\AppData\Local\Programs\Python\Python39\lib\site-packages\elevation\datasource.py", line 148, in ensure_tiles
return util.check_call_make(path, targets=['download'], variables=variables_items, **kwargs)

File "C:\Users\FALAH FAKHRI\AppData\Local\Programs\Python\Python39\lib\site-packages\elevation\util.py", line 88, in check_call_make
subprocess.check_call(cmd, shell=True)

File "C:\Users\FALAH FAKHRI\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)

CalledProcessError: Command 'make -C C:\Users\FALAH FAKHRI\AppData\Local\bopen\elevation\Cache\SRTM1 download ENSURE_TILES="N41/N41E012.tif"' returned non-zero exit status 1.