Support for netcdf4/hdf5 compression

[aidanheerdegen]

It would be great to be able to specify netCDF4 compression parameters when saving datasets.

If this is unlikely to be supported, can you suggest a reasonable work-around? I am assuming it would involve directly accessing a backend?

[clarkfitzg]

This seems useful. xray uses the netCDF4 library here, and they support it. In the meantime, you could always add a post processing step from the command line: http://www.unidata.ucar.edu/blogs/developer/en/entry/netcdf_compression.

[jhamman]

I don't see any reason why we couldn't support this. The difficulty is that the implementation will be different (or not possible) for difficult backends.

netCDF4 adds compression at the Variable level so we would have to think about how to implement this to our Dataset.to_netcdf method. Would we end up setting the compression level / type in each DataArray or would we add an argument to the to_netcdf method?

[shoyer]

This is actually already supported, though poorly documented (so it's basically unknown).

We seem to have some sort of bug in our documentation generation for recent versions, but in the v0.5.1 IO docs, you can see the encoding attribute at the end of the section on writing netCDFs:
http://xray.readthedocs.org/en/v0.5.1/io.html#netcdf

The way this works is that encoding on each data array stores a dictionary of options that is used when serializing that array to disk. It support most of the options in netCDF4-python's createVariable method, including chunksizes, zlib, scale_factor, add_offset, _FillValue and dtype. This metadata is automatically filled in when reading a file from disk, which means that in principle xray should roundtrip the encoding faithfully.

Because encoding is read in when files are opened, invalid encoding options are currently ignored when saving a file to disk. This means that the current API is not very user friendly.

So I'd like to extend this into a keyword argument encoding for the to_netcdf method. The keyword argument would expect a dictionary where the keys are variable names and the values are encoding parameters, and errors would be raised for invalid encoding options. Here's my branch for that feature:
https://github.com/shoyer/xray/tree/encoding-error-handling

[aidanheerdegen]

Brilliant. Thanks. I looked into the code but thought the encoding information was being stripped out.

So I've confirmed xray will round-trip fine. Shallow copies also round trip. Similarly making a new dataset from a variable with encoding information preserves that information and will output properly.

% tmp = xray.open_dataset('saved_on_disk_compressed.nc')
% tmp
<xray.Dataset>
Dimensions:         (time: 3, x: 2, y: 2)
Coordinates:
    reference_time  datetime64[ns] 2014-09-05
    lon             (x, y) float64 -99.83 -99.32 -99.79 -99.23
  * time            (time) datetime64[ns] 2014-09-06 2014-09-07 2014-09-08
    lat             (x, y) float64 42.25 42.21 42.63 42.59
  * x               (x) int64 0 1
  * y               (y) int64 0 1
Data variables:
    temperature     (x, y, time) float64 10.66 8.539 6.713 8.519 29.07 27.86 ...
    precipitation   (x, y, time) float64 0.3385 6.773 8.985 0.9651 0.1359 ...

% tmp.temperature.encoding
{'chunksizes': (2, 2, 3),
 'complevel': 5,
 'contiguous': False,
 'dtype': dtype('float64'),
 'fletcher32': False,
 'shuffle': True,
 'source': 'saved_on_disk_compressed.nc',
 'zlib': True}

% tmp2 = tmp
% tmp2.to_netcdf('saved_on_disk_comp_tmp2.nc')
% tmp3 = xray.open_dataset('saved_on_disk_comp_tmp2.nc')
% tmp3.temperature.encoding 
{'chunksizes': (2, 2, 3),
 'complevel': 5,
 'contiguous': False,
 'dtype': dtype('float64'),
 'fletcher32': False,
 'shuffle': True,
 'source': 'saved_on_disk_comp_xray.nc',
 'zlib': True}

Setting encoding dictionary works fine too (in this case copying from an existing variable):

% tmp4 = xray.DataArray(tmp.temperature.values).to_dataset(name='temperature')
% tmp4.temperature.encoding
{}
% tmp4.temperature.encoding = tmp.temperature.encoding                                                                              
% tmp4.temperature.encoding 
{'chunksizes': (2, 2, 3),
 'complevel': 5,
 'contiguous': False,
 'dtype': dtype('float64'),
 'fletcher32': False,
 'shuffle': True,
 'source': 'saved_on_disk_compressed.nc',
 'zlib': True}
% tmp4.to_netcdf('saved_on_disk_comp_tmp4.nc')
% tmp5 = xray.open_dataset('saved_on_disk_comp_tmp4.nc')
% tmp.temperature.encoding
{'chunksizes': (2, 2, 3),
 'complevel': 5,
 'contiguous': False,
 'dtype': dtype('float64'),
 'fletcher32': False,
 'shuffle': True,
 'source': 'saved_on_disk_compressed.nc',
 'zlib': True}

That will do nicely. Thanks.