Regression in createVariable between 1.5.8 and 1.6.0

[mrsalo]

• netcdf4 version 1.6.0
• Arch Linux and Rocky Linux 8

See the attached file to reproduce the behaviour, it works with 1.5.8, but does not with 1.6.0, but instead prints the following error message:

# v1.5.8
$ pip install --upgrade netcdf4==1.5.8 2&>1 > /dev/null; python regression.py
$
# v1.6.0
$ pip install --upgrade netcdf4==1.6.0 2&>1 > /dev/null; python regression.py
Traceback (most recent call last):
  File "/.../regression.py", line 21, in <module>
    main()
  File "/.../regression.py", line 10, in main
    cdf["data"].createVariable(
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4003, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: bad id or parameters or duplicate filter
$

# content of regression.py
import netCDF4


def main():
    cdf = netCDF4.Dataset("bug.nc", "w", format="NETCDF4", memory=None)
    cdf.createGroup("data")
    dim_size = 100
    cdf["data"].createDimension("number_tm_packets", dim_size)
    row_of_bytes = cdf["data"].createVLType("uint8", "row_of_bytes")
    cdf["data"].createVariable(
        varname="tm_packets",
        datatype=row_of_bytes,
        dimensions=("number_tm_packets",),
        zlib=True,
        shuffle=True,
        endian="native",
    )


if __name__ == '__main__':
    main()

[jswhit]

Compression does not work with variable-length types. Maybe this did not cause an error with netcdf-c 4.8.1 (which was used for the netcdf4-python 1.5.8 wheel), but it looks like it does for netcdf-c 4.9.0 (which is used by the netcdf4-python 1.6.0 wheel). The error message (which comes from the C lib) is somewhat confusing though.

See https://forum.hdfgroup.org/t/compression-of-variable-lenght-datatypes/1384

[mrsalo]

Thank you very much for the clarification, I was not aware of that.
Removing the zlib=True fixes the issue.
I opened an issue in netcdf-c to improve the error message: Unidata/netcdf-c#2447