h5netcdf fails to decode attribute coordinates.

[yt87]

What happened:
The engine h5netcdf fail to decode attribute coordinates.

What you expected to happen:
It should work.

Minimal Complete Verifiable Example:

# Put your MCVE code here
import xarray as xr
ds = xr.open_dataset('/tmp/x.nc', engine='h5netcdf')

========H5 coordinates ['x y']

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-481117dce7ff> in <module>
      1 import xarray as xr
      2 
----> 3 ds = xr.open_dataset('/tmp/x.nc', engine='h5netcdf')

~/miniconda3/envs/aws/lib/python3.7/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta)
    572 
    573     with close_on_error(store):
--> 574         ds = maybe_decode_store(store, chunks)
    575 
    576     # Ensure source filename always stored in dataset object (GH issue #2550)

~/miniconda3/envs/aws/lib/python3.7/site-packages/xarray/backends/api.py in maybe_decode_store(store, chunks)
    476             drop_variables=drop_variables,
    477             use_cftime=use_cftime,
--> 478             decode_timedelta=decode_timedelta,
    479         )
    480 

~/miniconda3/envs/aws/lib/python3.7/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
    596         drop_variables=drop_variables,
    597         use_cftime=use_cftime,
--> 598         decode_timedelta=decode_timedelta,
    599     )
    600     ds = Dataset(vars, attrs=attrs)

~/miniconda3/envs/aws/lib/python3.7/site-packages/xarray/conventions.py in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
    504             if "coordinates" in var_attrs:
    505                 coord_str = var_attrs["coordinates"]
--> 506                 var_coord_names = coord_str.split()
    507                 if all(k in variables for k in var_coord_names):
    508                     new_vars[k].encoding["coordinates"] = coord_str

AttributeError: 'numpy.ndarray' object has no attribute 'split'

Anything else we need to know?:
The test file was created from CDL:

netcdf x {
dimensions:
	x = 1 ;
	y = 1 ;
variables:
	int foo(y, x) ;
		string foo:coordinates = "x y" ;
data:

 foo =
  0 ;
}

The line ========H5 coordinates ['x y'] comes from me adding print statement on line 56 in function _read_attributes, file api/h5netcdf.py. Obviously the problem is caused by the attribute being a list instead of a string, as it is when netcdf4 is used.

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.7 (default, Mar 23 2020, 22:36:06)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 5.9.12-200.fc33.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.1

xarray: 0.16.2
pandas: 1.2.0
numpy: 1.19.2
scipy: 1.5.2
netCDF4: 1.4.2
pydap: None
h5netcdf: 0.8.1
h5py: 2.10.0
Nio: None
zarr: None
cftime: 1.3.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2021.01.0
distributed: 2021.01.0
matplotlib: 3.2.1
cartopy: 0.17.0
seaborn: None
numbagg: None
pint: None
setuptools: 51.1.2.post20210112
pip: 20.3.3
conda: None
pytest: 5.4.3
IPython: 7.19.0
sphinx: None

[kmuehlbauer]

@yt87 AFAIK the CDL doesn't declare attribute types specifically. This should work:

netcdf x {
dimensions:
	x = 1 ;
	y = 1 ;
variables:
	int foo(y, x) ;
		foo:coordinates = "x y" ;
data:

 foo =
  0 ;
}

[yt87]

Dropping string changes error to Unable to open file (file signature not found). This issue popped up while reading data from https://registry.opendata.aws/wrf-se-alaska-snap/

[kmuehlbauer]

Hmm, this worked in my case. How exactly did you create the netcdf file?

[kmuehlbauer]

If you want a quick fix to read the file, add kwarg decode_cf=False in open_dataset. Then fix the coordinate attribute and call xr.decode_cf(ds).

[yt87]

This is how I did it:

$ ncdump /tmp/x.nc
netcdf x {
dimensions:
	x = 1 ;
	y = 1 ;
variables:
	int foo(y, x) ;
		foo:coordinates = "x y" ;
data:

 foo =
  0 ;
}
$ rm x.nc 
$ ncgen -o x.nc < x.cdl
$ python -c "import xarray as xr; ds = xr.open_dataset('/tmp/x.nc', engine='h5netcdf'); print(ds)"

Engine netcdf4 works fine, with string or without.

My original code retrieving data from AWS:

import s3fs
import xarray as xr

s3 = s3fs.S3FileSystem(anon=True)
s3path = 's3://wrf-se-ak-ar5/gfdl/hist/daily/1988/WRFDS_1988-04-23.nc'

ds = xr.open_dataset(s3.open(s3path))
print(ds)

Adding decode_cf=False is a workaround. All attributes are arrays:

Attributes:
    contact:  ['rtladerjr@alaska.edu']
    info:     ['Alaska CASC']
    data:     ['Downscaled GFDL-CM3']
    format:   ['version 2']
    date:     ['Mon Jul  1 15:17:16 AKDT 2019']

[kmuehlbauer]

@yt87 Thanks for providing all this.

Could you please give -k 'nc4' as option to the call to ncgen and report back? I think this is a problem with data model.

[yt87]

You might be right. Adding -k nc4 works when string is removed from attribute specification. If it is present, the error is as before: AttributeError: 'numpy.ndarray' object has no attribute 'split'.

However, after changing my AWS script to

import s3fs
import xarray as xr

s3 = s3fs.S3FileSystem(anon=True)
s3path = 's3://wrf-se-ak-ar5/gfdl/hist/daily/1988/WRFDS_1988-04-23.nc'

ds = xr.open_dataset(s3.open(s3path), engine='scipy')
print(ds)

the error is TypeError: Error: None is not a valid NetCDF 3 file.

[kmuehlbauer]

Possibly related issue: h5netcdf/h5netcdf#27

Anyway, I think I've found the root-cause of the issue. So with netcdf4-python it is possible to specify writing of attributes as NC_STRING string attributes.

The following will create a netcdf file which is readable with current xarray and h5netcdf backend.

import netCDF4 as nc
import xarray as xr
rootgrp = nc.Dataset("test_nc4_attrs.nc", mode="w", format="NETCDF4")
# rootgrp.set_ncstring_attrs(True)
x = rootgrp.createDimension("x", 1)
y = rootgrp.createDimension("y", 1)
foo = rootgrp.createVariable("foo", "i4", ("y", "x"))
foo.coordinates = "y x"
foo[0, 0] = 0
rootgrp.close()
ds = xr.open_dataset("test_nc4_attrs.nc", engine="h5netcdf")
print(ds)
print(ds.foo)
ds.close()

If you uncomment the one line, this creates netcdf file which breaks with the error @yt87 showed above. If you try to open the file with decode_cf=False and write it back using to_netcdf the same error occurs.

h5dump before uncomment

 ATTRIBUTE "coordinates" {
         DATATYPE  H5T_STRING {
            STRSIZE 3;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "y x"
         }
      }

h5dump after uncomment

ATTRIBUTE "coordinates" {
         DATATYPE  H5T_STRING {
            STRSIZE H5T_VARIABLE;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_UTF8;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
         DATA {
         (0): "y x"
         }
      }

Another thing I found, that this only accounts for coordinate attributes. All other attributes are read correctly AFAIKT. So with this in mind I would consider that a bug. The assumption that attributes are always scalar doesn't hold true.

And one more thing, also roundtripping in netcdf4 does not keep the NC_STRING type:

import netCDF4 as nc
import xarray as xr
rootgrp = nc.Dataset("test_nc4_attrs.nc", mode="w", format="NETCDF4")
rootgrp.set_ncstring_attrs(True)
x = rootgrp.createDimension("x", 1)
y = rootgrp.createDimension("y", 1)
foo = rootgrp.createVariable("foo", "i4", ("y", "x"))
rootgrp.coordinates = "z y x"
foo.coordinates = "y x"
foo[0, 0] = 0
rootgrp.close()

ds = xr.open_dataset("test_nc4_attrs.nc", engine="netcdf4")
ds.to_netcdf("test_nc4_attrs_out.nc")
ds.close()

ncdump test_nc4_attrs.nc

netcdf test_nc4_attrs {
dimensions:
	x = 1 ;
	y = 1 ;
variables:
	int foo(y, x) ;
		string foo:coordinates = "y x" ;

// global attributes:
		string :coordinates = "z y x" ;
data:

 foo =
  0 ;
}

ncdump test_nc4_attrs_out.nc

netcdf test_nc4_attrs_out {
dimensions:
	y = 1 ;
	x = 1 ;
variables:
	int foo(y, x) ;
		foo:coordinates = "y x" ;
data:

 foo =
  0 ;
}

So changes are needed in several places to check if the attribute is scalar or array, not only for coordinates attributes. Thoughts?

[kmuehlbauer]

@yt87 This works for me with latest h5netcdf version. There have been several fixes regarding attribute reading. The issue might be closed after confirming.

[dcherian]

Thanks @kmuehlbauer .

@yt87 please reopen if you still have trouble after upgrading h5netcdf