Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download function for JAXA's AW3D30 DEM #734

merged 16 commits into from Apr 10, 2019
Changes from 7 commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.


Just for now

@@ -22,6 +22,11 @@ Enhancements
By `Beatriz Recinos <>`_.
- Added a mechanism to add custom MB data to OGGM (:issue:`724`).
By `Fabien Maussion <>`_.
- The ALOS Global Digital Surface Model "ALOS World 3D - 30m" DEM from JAXA can
now be used as alternative DEM within OGGM.
`See our tutorial <>`_ on how to
set an alternative DEM (:pull:`734`).
By `Matthias Dusch <>`_.
This conversation was marked as resolved by matthiasdusch

This comment has been minimized.

Copy link

fmaussion Apr 8, 2019


Nit: space missing in front of "by"

Bug fixes
@@ -45,6 +45,7 @@
HttpDownloadError, HttpContentTooShortError)
from oggm.utils._workflow import robust_tar_extract

# Module logger
logger = logging.getLogger('.'.join(__name__.split('.')[:-1]))
@@ -341,7 +342,7 @@ def _requests_urlretrieve(url, path, reporthook, auth=None):
raise HttpContentTooShortError()

def _classic_urlretrieve(url, path, reporthook, auth=None):
def _classic_urlretrieve(url, cache_path, reporthook, auth=None):
This conversation was marked as resolved by matthiasdusch

This comment has been minimized.

Copy link

TimoRoth Apr 9, 2019


I just fixed this in the opposite way myself, so this change here needs undone or it will introduce the same issue again, just reversed.

"""Thin wrapper around pythons urllib urlretrieve

@@ -838,6 +839,50 @@ def _download_arcticdem_from_cluster_unlocked():
return outpath

def _download_aw3d30_file(zone):
with _get_download_lock():
return _download_aw3d30_file_unlocked(zone)

def _download_aw3d30_file_unlocked(fullzone):
"""Checks if the AW3D30 data is in the directory and if not, download it.

# extract directory
tmpdir = cfg.PATHS['tmp_dir']

# tarfiles are extracted in directories per each tile
tile = fullzone.split('/')[1]
demfile = os.path.join(tmpdir, tile, tile + '_AVE_DSM.tif')

# check if extracted file exists already
if os.path.exists(demfile):
return demfile

# Did we download it yet?
ftpfile = (''
+ fullzone + '.tar.gz')
dest_file = file_downloader(ftpfile)
This conversation was marked as resolved by TimoRoth

This comment has been minimized.

Copy link

matthiasdusch Apr 4, 2019

Author Member

this doesn't work right now

This comment has been minimized.

Copy link

TimoRoth Apr 4, 2019


requests does not support downloading from ftp. I searched quite a bit on how to do it, and only found ugly hacks and conclusions that it's an old protocol and they want people to use https instead.

This comment has been minimized.

Copy link

fmaussion Apr 4, 2019


This is a clear incentive to solve the license issue. Since we have the files on cluster we could either:

  • make them open and download from there
  • hide them behind a password and use the same technique as for Tandem-X with a local credential file. Then we can control who gets the password (my preferred approach from now)

# None means we tried hard but we couldn't find it
if not dest_file:
return None

# ok we have to extract it

if not os.path.exists(demfile):
dempath = os.path.dirname(demfile)
robust_tar_extract(dest_file, dempath)

# See if we're good, don't overfill the tmp directory
assert os.path.exists(demfile)
# this tarfile contains several files
for file in os.listdir(dempath):
cfg.get_lru_handler(tmpdir).append(os.path.join(dempath, file))
return demfile

def _get_centerline_lonlat(gdir):
"""Quick n dirty solution to write the centerlines as a shapefile"""

@@ -961,6 +1006,50 @@ def tandem_zone(lon_ex, lat_ex):
return list(sorted(set(zones)))

def _aw3d30_path(lon_tile, lat_tile):

# OK we have a proper tile now

# Folders are sorted with N E S W in 5 degree steps
# But in N and E the lower boundary is indicated
# e.g. N060 contains N060 - N064
# e.g. E000 contains E000 - E004
# but S and W indicate the upper boundary:
# e.g. S010 contains S006 - S010
# e.g. W095 contains W091 - W095

# get letters
ns = 'S' if lat_tile < 0 else 'N'
ew = 'W' if lon_tile < 0 else 'E'

# get lat/lon
lon = abs(5 * np.floor(lon_tile/5))
lat = abs(5 * np.floor(lat_tile/5))

folder = '%s%.3d%s%.3d' % (ns, lat, ew, lon)
filename = '%s%.3d%s%.3d' % (ns, abs(lat_tile), ew, abs(lon_tile))

# Final path
out = folder + '/' + filename
return out

def aw3d30_zone(lon_ex, lat_ex):
"""Returns a list of AW3D30 zones covering the desired extent.

# Files are one by one tiles, so lets loop over them
lon_tiles = np.arange(np.floor(lon_ex[0]), np.ceil(lon_ex[1]+1e-9),
lat_tiles = np.arange(np.floor(lat_ex[0]), np.ceil(lat_ex[1]+1e-9),
zones = []
for lon in lon_tiles:
for lat in lat_tiles:
zones.append(_aw3d30_path(lon, lat))
return list(sorted(set(zones)))

def arcticdem_zone(lon_ex, lat_ex):
"""Returns a list of Arctic-DEM zones covering the desired extent.
@@ -1627,6 +1716,7 @@ def get_topo_file(lon_ex, lat_ex, rgi_region=None, rgi_subregion=None,
- 'ASTER' : ASTER data
- 'TANDEM' :
- 'AW3D30' :
@@ -1693,6 +1783,14 @@ def get_topo_file(lon_ex, lat_ex, rgi_region=None, rgi_subregion=None,
source_str = source

# AW3D30 - ALOS Global Digital Surface Model 3D 30m from JAXA
if source == 'AW3D30':
zones = aw3d30_zone(lon_ex, lat_ex)
sources = []
for z in zones:
source_str = source

# Anywhere else on Earth we check for DEM3, ASTER, or SRTM
# exceptional test for eastern russia:
east_max = np.min(lat_ex) > 59 and np.min(lon_ex) > 170
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.