How to reduce the output size with to_netcdf?

[simon3122]

Hello,

The options of the ds.to_netcdf method allow several netcdf modes and a control of several options (missing values, format...).

Our current options are as follows:

ds.to_netcdf(filenam,
'w',
'NETCDF4',
'netcdf4',
encoding={ vkey2:{'dtype': float32,'_FillValue':0}})

Whereas the input size is 6 GiB / variable, the output size reaches 23 GiB / variable.

What is the best strategy in the options to reduce the netcdf output size, other than changing the format (in my exercice we have float32)?
In particular, is there some interest in modifying the deprecated "missing_values" attribute?

Thank you in advance,

[fmaussion]

NetCDF and xarray support lossy compression (extremely efficient and fast with the cost of numerical precision loss) or gzip compression (without precision loss but with slower I/O - especially when reading chunks of data). You can have a look at the documentation about NetCDF I/O here.

[marcosrdac]

How do I get lossy compression? I could not find it on the documentation :(

[dcherian]

Please read the documentation

[marcosrdac]

Thanks, I thought there were some methods to choose from or something like that. For future readers, scale_factor seems to be used to control compression loss.

[dcherian]

You can also use zlib and complevel

[marcosrdac]

You can also use zlib and complevel

Just making it clear: those would configure lossless compression of netcdf4 lib, not lossy compression.