Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download function for JAXA's AW3D30 DEM #734

Merged
merged 16 commits into from Apr 10, 2019
Merged
Changes from 5 commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -20,6 +20,11 @@ Enhancements
mass-conservation inversion (`inversion.find_inversion_calving`). This
is still in experimentation phase! (:pull:`720`).
By `Beatriz Recinos <https://github.com/bearecinos>`_.
- The ALOS Global Digital Surface Model "ALOS World 3D - 30m" DEM from JAXA can
now be used as alternative DEM within OGGM.
`See our tutorial <http://edu.oggm.org/en/latest/oggm_tuto.html>`_ on how to
set an alternative DEM (:pull:`734`).
By `Matthias Dusch <https://github.com/matthiasdusch>`_.
This conversation was marked as resolved by matthiasdusch

This comment has been minimized.

Copy link
@fmaussion

fmaussion Apr 8, 2019

Member

Nit: space missing in front of "by"


Bug fixes
~~~~~~~~~
@@ -45,6 +45,7 @@
DownloadVerificationFailedException,
DownloadCredentialsMissingException,
HttpDownloadError, HttpContentTooShortError)
from oggm import utils
This conversation was marked as resolved by matthiasdusch

This comment has been minimized.

Copy link
@fmaussion

fmaussion Apr 8, 2019

Member

There is a risk of circular dependencies here - utils is in fact three modules including this one. Remove this line and import directly in the function itself with from oggm.utils._workflow import robust_tar_extract

This comment has been minimized.

Copy link
@fmaussion

fmaussion Apr 8, 2019

Member

(reason: not so good design of a huge utils module...)


# Module logger
logger = logging.getLogger('.'.join(__name__.split('.')[:-1]))
@@ -341,7 +342,7 @@ def _requests_urlretrieve(url, path, reporthook, auth=None):
raise HttpContentTooShortError()


def _classic_urlretrieve(url, path, reporthook, auth=None):
def _classic_urlretrieve(url, cache_path, reporthook, auth=None):
This conversation was marked as resolved by matthiasdusch

This comment has been minimized.

Copy link
@TimoRoth

TimoRoth Apr 9, 2019

Contributor

I just fixed this in the opposite way myself, so this change here needs undone or it will introduce the same issue again, just reversed.

"""Thin wrapper around pythons urllib urlretrieve
"""

@@ -838,6 +839,50 @@ def _download_arcticdem_from_cluster_unlocked():
return outpath


def _download_aw3d30_file(zone):
with _get_download_lock():
return _download_aw3d30_file_unlocked(zone)


def _download_aw3d30_file_unlocked(fullzone):
"""Checks if the AW3D30 data is in the directory and if not, download it.
"""

# extract directory
tmpdir = cfg.PATHS['tmp_dir']
mkdir(tmpdir)

# tarfiles are extracted in directories per each tile
tile = fullzone.split('/')[1]
demfile = os.path.join(tmpdir, tile, tile + '_AVE_DSM.tif')

# check if extracted file exists already
if os.path.exists(demfile):
return demfile

# Did we download it yet?
ftpfile = ('ftp://ftp.eorc.jaxa.jp/pub/ALOS/ext1/AW3D30/release_v1804/'
+ fullzone + '.tar.gz')
dest_file = file_downloader(ftpfile)
This conversation was marked as resolved by TimoRoth

This comment has been minimized.

Copy link
@matthiasdusch

matthiasdusch Apr 4, 2019

Author Member

this doesn't work right now

This comment has been minimized.

Copy link
@TimoRoth

TimoRoth Apr 4, 2019

Contributor

requests does not support downloading from ftp. I searched quite a bit on how to do it, and only found ugly hacks and conclusions that it's an old protocol and they want people to use https instead.

This comment has been minimized.

Copy link
@fmaussion

fmaussion Apr 4, 2019

Member

This is a clear incentive to solve the license issue. Since we have the files on cluster we could either:

  • make them open and download from there
  • hide them behind a password and use the same technique as for Tandem-X with a local credential file. Then we can control who gets the password (my preferred approach from now)

# None means we tried hard but we couldn't find it
if not dest_file:
return None

# ok we have to extract it

if not os.path.exists(demfile):
dempath = os.path.dirname(demfile)
utils.robust_tar_extract(dest_file, dempath)

# See if we're good, don't overfill the tmp directory
assert os.path.exists(demfile)
# this tarfile contains several files
for file in os.listdir(dempath):
cfg.get_lru_handler(tmpdir).append(os.path.join(dempath, file))
return demfile


def _get_centerline_lonlat(gdir):
"""Quick n dirty solution to write the centerlines as a shapefile"""

@@ -961,6 +1006,50 @@ def tandem_zone(lon_ex, lat_ex):
return list(sorted(set(zones)))


def _aw3d30_path(lon_tile, lat_tile):

# OK we have a proper tile now

# Folders are sorted with N E S W in 5 degree steps
# But in N and E the lower boundary is indicated
# e.g. N060 contains N060 - N064
# e.g. E000 contains E000 - E004
# but S and W indicate the upper boundary:
# e.g. S010 contains S006 - S010
# e.g. W095 contains W091 - W095

# get letters
NS = 'S' if lat_tile < 0 else 'N'
EW = 'W' if lon_tile < 0 else 'E'
This conversation was marked as resolved by matthiasdusch

This comment has been minimized.

Copy link
@fmaussion

fmaussion Apr 8, 2019

Member

nit: keep ns and ew lower case


# get lat/lon
lon = abs(5 * np.floor(lon_tile/5))
lat = abs(5 * np.floor(lat_tile/5))

folder = '%s%.3d%s%.3d' % (NS, lat, EW, lon)
file = '%s%.3d%s%.3d' % (NS, abs(lat_tile), EW, abs(lon_tile))
This conversation was marked as resolved by matthiasdusch

This comment has been minimized.

Copy link
@fmaussion

fmaussion Apr 8, 2019

Member

nit: file is a python dedicated name - use fpath instead (or similar)


# Final path
out = folder + '/' + file
return out


def aw3d30_zone(lon_ex, lat_ex):
"""Returns a list of AW3D30 zones covering the desired extent.
"""

# Files are one by one tiles, so lets loop over them
lon_tiles = np.arange(np.floor(lon_ex[0]), np.ceil(lon_ex[1]+1e-9),
dtype=np.int)
lat_tiles = np.arange(np.floor(lat_ex[0]), np.ceil(lat_ex[1]+1e-9),
dtype=np.int)
zones = []
for lon in lon_tiles:
for lat in lat_tiles:
zones.append(_aw3d30_path(lon, lat))
return list(sorted(set(zones)))


def arcticdem_zone(lon_ex, lat_ex):
"""Returns a list of Arctic-DEM zones covering the desired extent.
"""
@@ -1627,6 +1716,7 @@ def get_topo_file(lon_ex, lat_ex, rgi_region=None, rgi_subregion=None,
- 'ASTER' : ASTER data
- 'TANDEM' : https://geoservice.dlr.de/web/dataguide/tdm90/
- 'ARCTICDEM' : https://www.pgc.umn.edu/data/arcticdem/
- 'AW3D30' : https://www.eorc.jaxa.jp/ALOS/en/aw3d30
Returns
-------
@@ -1693,6 +1783,14 @@ def get_topo_file(lon_ex, lat_ex, rgi_region=None, rgi_subregion=None,
sources.append(_download_tandem_file(z))
source_str = source

# AW3D30 - ALOS Global Digital Surface Model 3D 30m from JAXA
if source == 'AW3D30':
zones = aw3d30_zone(lon_ex, lat_ex)
sources = []
for z in zones:
sources.append(_download_aw3d30_file(z))
source_str = source

# Anywhere else on Earth we check for DEM3, ASTER, or SRTM
# exceptional test for eastern russia:
east_max = np.min(lat_ex) > 59 and np.min(lon_ex) > 170
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.