Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support new netcdf4 1.6.0 compression arguments #6929

Closed
markelg opened this issue Aug 18, 2022 · 2 comments · Fixed by #6981
Closed

Support new netcdf4 1.6.0 compression arguments #6929

markelg opened this issue Aug 18, 2022 · 2 comments · Fixed by #6981

Comments

@markelg
Copy link
Contributor

markelg commented Aug 18, 2022

Is your feature request related to a problem?

When using the netcdf4 engine, I am not able to use the new "compression" argument to choose a compression scheme different from zlib in the encoding.

        if raise_on_invalid:
            invalid = [k for k in encoding if k not in valid_encodings]
            if invalid:
>               raise ValueError(
                    f"unexpected encoding parameters for {backend!r} backend: {invalid!r}. Valid "
                    f"encodings are: {valid_encodings!r}"
                )
E               ValueError: unexpected encoding parameters for 'netCDF4' backend: ['compression']. Valid encodings are: {'fletcher32', 'zlib', 'contiguous', 'dtype', 'least_significant_digit', 'shuffle', 'complevel', '_FillValue', 'chunksizes'}

../../../../netCDF4_.py:279: ValueError

Furthermore, according to the release notes of 1.6.0, zlib argument is to be deprecated:

* add 'compression' kwarg to createVariable to enable new compression
   functionality in netcdf-c 4.9.0.  'None','zlib','szip','zstd','bzip2'
   'blosc_lz','blosc_lz4','blosc_lz4hc','blosc_zlib' and 'blosc_zstd'
   are currently supported. 'blosc_shuffle',
   'szip_mask' and 'szip_pixels_per_block' kwargs also added.
   compression='zlib' is equivalent to (the now deprecated) zlib=True.
   If the environment variable NETCDF_PLUGIN_DIR is set to point to the
   directory with the compression plugin lib__nc* files, then the compression plugins will
   be installed within the package and be automatically available (the binary
   wheels have this).  Otherwise, the environment variable HDF5_PLUGIN_PATH
   needs to be set at runtime  to point to plugins in order to use the new compression
   options.

I am using the last versions

xarray: 2022.6.0
pandas: 1.4.2
numpy: 1.22.4
scipy: None
netCDF4: 1.6.0
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.6.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2022.6.0
distributed: 2022.6.0
matplotlib: 3.5.2
cartopy: None
seaborn: None
numbagg: None
fsspec: 2022.5.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 62.6.0
pip: 22.1.2
conda: None
pytest: 7.1.2
IPython: 8.4.0
sphinx: 5.1.1

Describe the solution you'd like

Update the netcdf4 backend to support these arguments. Should not be too difficult.

Describe alternatives you've considered

No response

Additional context

I can try to do this myself, it does not look hard.

@markelg markelg changed the title Support new netcdf4 1.60 compresison arguments Support new netcdf4 1.6.0 compresison arguments Aug 18, 2022
@markelg markelg changed the title Support new netcdf4 1.6.0 compresison arguments Support new netcdf4 1.6.0 compression arguments Aug 18, 2022
@dcherian
Copy link
Contributor

I think you just need to add compression here:

valid_encodings = {
"zlib",
"complevel",
"fletcher32",
"contiguous",
"chunksizes",
"shuffle",
"_FillValue",
"dtype",
}

We'll want a test too in test_backends.py

@markelg
Copy link
Contributor Author

markelg commented Sep 2, 2022

Thank you @dcherian. I opened a pull request, please take a look. I added "compression" to the list, but it will be removed if a version of netCDF4 below 1.6.0 is detected. It may be a good idea also to check for the latest versions of netcdf-c.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants