new vsis3 features for NETCDF group datasets not supported in conda #376

dbekaert · 2020-04-16T04:24:03Z

Issue:
We use conda for teaching GDAL and currently can't use the new vsis3 features for NETCDF group datasets. The netcdf dataset with group in S3 are al recognized using the HDF5 and not the NETCDF reader. (similar report in #323)

Expected behavior (local run):

 gdalinfo S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc
Driver: netCDF/Network Common Data Format

S3 vsi behavior:

from osgeo import gdal 
gdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','cookies.txt') 
gdal.SetConfigOption('GDAL_HTTP_COOKIEJAR', 'cookies.txt') 
results = gdal.Info("/vsicurl/https://grfn.asf.alaska.edu/door/download/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc")

results show:

Driver: HDF5/Hierarchical Data Format Release 5
Files: /vsicurl/https://grfn.asf.alaska.edu/door/download/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc

Environment:
Problem persists on:

linux: x86_64 x86_64 x86_64 GNU/Linux with gdal 3.0.4 installed using conda
mac: Darwin MT-101939 17.7.0 Darwin Kernel Version 17.7.0: Tue Feb 18 22:51:29 PST 2020; root:xnu-4570.71.73~1/RELEASE_X86_64 x86_64 with gdal 3.0.4 installed using conda

Details about conda and system ( conda info ):

On MacOS

     active environment : ariatools_kite
    active env location : /Users/dbekaert/Software/python/miniconda37/envs/ariatools_kite
            shell level : 2
       user config file : /Users/dbekaert/.condarc
 populated config files : /Users/dbekaert/.condarc
          conda version : 4.6.14
    conda-build version : not installed
         python version : 3.7.3.final.0
       base environment : /Users/dbekaert/Software/python/miniconda37  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/osx-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/dbekaert/Software/python/miniconda37/pkgs
                          /Users/dbekaert/.conda/pkgs
       envs directories : /Users/dbekaert/Software/python/miniconda37/envs
                          /Users/dbekaert/.conda/envs
               platform : osx-64
             user-agent : conda/4.6.14 requests/2.21.0 CPython/3.7.3 Darwin/17.7.0 OSX/10.13.6
                UID:GID : 715963585:703763885
             netrc file : /Users/dbekaert/.netrc
           offline mode : False

On Linux

        active environment : ARIA-tools
    active env location : /u/leffe0/ssangha/tools/conda_installation/stable_feb9_2020/envs/ARIA-tools
            shell level : 1
       user config file : /home/dbekaert/.condarc
 populated config files : 
          conda version : 4.8.2
    conda-build version : not installed
         python version : 3.7.4.final.0
       virtual packages : __cuda=10.1
                          __glibc=2.17
       base environment : /u/leffe0/ssangha/tools/conda_installation/stable_feb9_2020  (read only)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /u/leffe0/ssangha/tools/conda_installation/stable_feb9_2020/pkgs
                          /home/dbekaert/.conda/pkgs
       envs directories : /home/dbekaert/.conda/envs
                          /u/leffe0/ssangha/tools/conda_installation/stable_feb9_2020/envs
               platform : linux-64
             user-agent : conda/4.8.2 requests/2.12.5 CPython/3.7.4 Linux/3.10.0-957.21.3.el7.x86_64 centos/7.6.1810 glibc/2.17
                UID:GID : 44067:59990
             netrc file : /home/dbekaert/.netrc
           offline mode : False

The text was updated successfully, but these errors were encountered:

gillins · 2020-04-16T04:32:34Z

See https://gdal.org/drivers/raster/netcdf.html#vsi-virtual-file-system-api-support

Does anyone know what version of the kernel we build against?

piyushrpt · 2020-04-16T05:01:14Z

We tried to fix this in #330
The latest build log says it was built with userfaultfd support - https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/137713/logs/26.

Still fails on a new linux kernel

uname -a 
5.3.0-7642-generic

Still fails on it - reports HDF5

piyushrpt · 2020-04-16T05:49:56Z

gdalmanage identifies the driver correctly but not gdalinfo

GDAL_HTTP_COOKIEFILE=asf.txt GDAL_HTTP_COOKIEJAR=asf.txt CPL_DEBUG=ON gdalmanage identify /vsicurl/https://grfn.asf.alaska.edu/door/download/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc
HTTP: libcurl/7.65.3 OpenSSL/1.1.1d zlib/1.2.5 libssh2/1.8.2
HTTP: GDAL was built against curl 7.64.1, but is running against 7.65.3.
VSICURL: Effective URL: https://dlz0dhr6g6ukw.cloudfront.net/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc?userid=piyush&Expires=1587016143&Signature=dal14OxjGrZxglgTFYckZlJMqe6571IByncAWUY5qfgGdU5CjAD6PNSpGYuiMtIVQJ1gz90B9mqN8f51v8OCY4Awc~CHIlStHFSS1J9nLkl6djWqolM1UOgWBId6ELTYyw5YELhchBBF2BIehwCUbKlz-MoRmfAImgNOiwq-8G0_&Key-Pair-Id=APKAINVNJF4BDB5SS5QQ
VSICURL: GetFileSize(https://grfn.asf.alaska.edu/door/download/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc)=58803964  response_code=200
VSICURL: Downloading 0-16383 (https://grfn.asf.alaska.edu/door/download/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc)...
VSICURL: Got response_code=206
VSICURL: Effective URL: https://dlz0dhr6g6ukw.cloudfront.net/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc?userid=piyush&Expires=1587016143&Signature=dal14OxjGrZxglgTFYckZlJMqe6571IByncAWUY5qfgGdU5CjAD6PNSpGYuiMtIVQJ1gz90B9mqN8f51v8OCY4Awc~CHIlStHFSS1J9nLkl6djWqolM1UOgWBId6ELTYyw5YELhchBBF2BIehwCUbKlz-MoRmfAImgNOiwq-8G0_&Key-Pair-Id=APKAINVNJF4BDB5SS5QQ
/vsicurl/https://grfn.asf.alaska.edu/door/download/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc: netCDF

dbekaert · 2020-04-16T19:20:32Z

@rouault just pushed a fix into gdal 2.4 and 3 branches. GDAL: OSGeo/gdal#2412

I am not sure on the pathway and time-frame before this might get picked up in the gdal conda. Any thoughts @gillins?

rouault · 2020-04-16T19:39:51Z

Note: GDAL 3.1 should be released within a few weeks. RC1 planned for April 27th

kmuehlbauer · 2020-04-16T19:50:41Z

@rouault Thanks Even!

@dbekaert @piyushrpt If this is a pressing issue, we could think about patching current feedstock. But if the next GDAL release is just ahead, it seems not worth the action.

dbekaert · 2020-04-16T20:47:09Z

We can wait a few weeks, allows for the through testing and integration.
I suspect that would be like end of May or so.

xylar · 2021-11-09T09:08:31Z

@dbekaert, was this addressed when we updated to 3.1? We're now at 3.3.3 so seems like it's time to revisit.

rouault · 2022-11-19T12:12:12Z

To be closed by someone with permissions to do so as working with recent GDAL versions:

(myenv) $ gdalinfo --version
GDAL 3.6.0, released 2022/11/06
(myenv) $ gdalinfo /vsicurl/http://download.osgeo.org/gdal/data/netcdf/utm-big-chunks.nc
Driver: netCDF/Network Common Data Format
Files: /vsicurl/http://download.osgeo.org/gdal/data/netcdf/utm-big-chunks.nc
Size is 3072, 3072
Origin = (440720.000000000000000,3751320.000000000000000)
Pixel Size = (10.000000000000000,-10.000000000000000)
Metadata:
  Band1#long_name=GDAL Band Number 1
  Band1#_FillValue=0
  NC_GLOBAL#CDI=Climate Data Interface version 1.6.0 (http://code.zmaw.de/projects/cdi)
  NC_GLOBAL#CDO=Climate Data Operators version 1.6.0 (http://code.zmaw.de/projects/cdo)
  NC_GLOBAL#Conventions=CF-1.5
  NC_GLOBAL#GDAL=GDAL 1.10.0, released 2013/04/13
  NC_GLOBAL#history=Tue Apr 16 16:07:46 2013: cdo -f nc4 -z zip -k auto copy utmbig.nc utmbig2.nc
Tue Apr 16 16:06:59 2013: GDAL Create( utmbig.nc, ... )
  x#axis=X
  x#long_name=x coordinate of projection
  x#standard_name=grid_longitude
  x#units=m
  y#axis=Y
  y#long_name=y coordinate of projection
  y#standard_name=grid_latitude
  y#units=m
Corner Coordinates:
Upper Left  (  440720.000, 3751320.000) 
Lower Left  (  440720.000, 3720600.000) 
Upper Right (  471440.000, 3751320.000) 
Lower Right (  471440.000, 3720600.000) 
Center      (  456080.000, 3735960.000) 
Band 1 Block=1536x1536 Type=Byte, ColorInterp=Undefined
  NoData Value=0
  Metadata:
    long_name=GDAL Band Number 1
    NETCDF_VARNAME=Band1
    _FillValue=0

dbekaert mentioned this issue Apr 16, 2020

Example of using vsicurl to GUNW products aria-tools/ARIA-tools#27

Closed

dbekaert changed the title ~~new vsis3 features for NETCDF group datasetsnot supported in conda~~ new vsis3 features for NETCDF group datasets not supported in conda Apr 16, 2020

piyushrpt mentioned this issue Apr 16, 2020

vsicurl and netcdf inconsistency OSGeo/gdal#2412

Closed

dbekaert mentioned this issue Apr 18, 2020

Circumvent netcdf4 driver to parse metadata aria-tools/ARIA-tools#141

Merged

akrherz closed this as completed Nov 19, 2022

scottstanie mentioned this issue Dec 27, 2023

rasterio does not understand netcdf url format when using Mac Arm rasterio/rasterio#2868

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new vsis3 features for NETCDF group datasets not supported in conda #376

new vsis3 features for NETCDF group datasets not supported in conda #376

dbekaert commented Apr 16, 2020 •

edited

gillins commented Apr 16, 2020

piyushrpt commented Apr 16, 2020 •

edited

piyushrpt commented Apr 16, 2020

dbekaert commented Apr 16, 2020

rouault commented Apr 16, 2020

kmuehlbauer commented Apr 16, 2020

dbekaert commented Apr 16, 2020

xylar commented Nov 9, 2021

rouault commented Nov 19, 2022

new vsis3 features for NETCDF group datasets not supported in conda #376

new vsis3 features for NETCDF group datasets not supported in conda #376

Comments

dbekaert commented Apr 16, 2020 • edited

gillins commented Apr 16, 2020

piyushrpt commented Apr 16, 2020 • edited

piyushrpt commented Apr 16, 2020

dbekaert commented Apr 16, 2020

rouault commented Apr 16, 2020

kmuehlbauer commented Apr 16, 2020

dbekaert commented Apr 16, 2020

xylar commented Nov 9, 2021

rouault commented Nov 19, 2022

dbekaert commented Apr 16, 2020 •

edited

piyushrpt commented Apr 16, 2020 •

edited