IO fails with complex datatypes

[arsenovic]

i am exploring the use of xarray as a back-end for some objects in the scikit-rf project.
this, and other electrical engineering applications require support for complex data-types.

• is there a [non-pickle] serialization that works with this?
• i dont get an error but it seems to loose the data?

https://gist.github.com/613be985bcf0001ebd05

( i have xarray 0.7.0)

[shoyer]

This looks bad! I was pretty sure we had tested this with h5netcdf. I'll take a look later... what version of h5py are you using?

[arsenovic]

2.5.0-np19py27_3 from conda

[shoyer]

OK, I can reproduce this. The issue seems to be that xarray uses the netCDF4 package as the default backend for reading netCDF4/HDF5 files, and netCDF4 seems to silently drop these variables with a compound dtype, even though allegedly it supports them.

The work around is to use engine='h5netcdf' explicitly when opening these netCDF files:

In [35]: ntwk_opened = xr.open_dataset('ring_slot.nc', engine='h5netcdf')

In [36]: ntwk_opened
Out[36]:
<xarray.Dataset>
Dimensions:  (f: 201, m: 2, n: 2)
Coordinates:
  * f        (f) float64 75.0 75.17 75.35 75.53 75.7 75.88 76.05 76.22 76.4 ...
  * m        (m) int64 0 1
  * n        (n) int64 0 1
Data variables:
    s        (f, m, n) complex128 (-0.503723180993+0.457844804761j) ...
    z0       (f, m) complex128 (50+0j) (50+0j) (50+0j) (50+0j) (50+0j) ...
Attributes:
    name: ring slot

[arsenovic]

excellent. thanks for looking into it, this work around is sufficient for my needs fyi.

[alexamici]

I add to this issue as it refers to general IO failures. Saving a Dataset that contains DataArray with complex data-type fails on all netCDF4 engines:

dataset = xr.Dataset({'data': np.zeros((2,), dtype='complex128')})
dataset.to_netcdf('tmp.nc', engine='netcdf4')
[...]
TypeError: illegal primitive data type, must be one of dict_keys(['i8', 'f4', 'f8', 'i2', 'u2', 'i1', 'i4', 'S1', 'u8', 'u1', 'u4']), got complex128

and

dataset.to_netcdf('tmp.nc', engine='h5netcdf')
[...]
TypeError: Object dtype dtype('O') has no native HDF5 equivalent

[shoyer]

@alexamici Something else must have gone wrong here -- your example works for me:

In [9]: dataset = xr.Dataset({'data': np.zeros((2,), dtype='complex128')})

In [10]: dataset
Out[10]:
<xarray.Dataset>
Dimensions:  (data: 2)
Coordinates:
  * data     (data) complex128 0j 0j
Data variables:
    *empty*

In [11]: dataset.to_netcdf('tmp.nc', engine='h5netcdf')

Maybe you need to update to a newer version of h5py? (Just a guess)

[alexamici]

I reinstalled a bunch of stuff and I can confirm my example above works for me now using engine='h5netcdf'. It still fails with the default engine.

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity
If this issue remains relevant, please comment here; otherwise it will be marked as closed automatically