Compare with grib2 format

[impredicative]

grib2 is a format commonly used by NWS for compressing spatially correlated weather forecast data, among other uses. I am curious how grib2 and zfp would compare both in theory and in practice, and if a hybrid would make sense. Thanks.

[lindstro]

My understanding of GRIB2 is that it's based on quantizing floating-point data to integers (following quantization levels determined to be appropriate for each physical quantity) and then applying JPEG2000, PNG, or other standard image compression algorithms to 2D slices. Both JPEG2000 and PNG are limited in the precision and data dimensionality they support. The JPEG2000 spec does have extensions for compressing 3D data, though as far as I know few tools support Part 10 of the spec, and there's to my knowledge no support for 3D + time. In terms of error tolerances, GRIB2 supports bounded errors only when used in conjunction with lossless compression. Finally, serial zfp is 1-2 orders of magnitude faster than JPEG2000 and also has support for parallel compression and other features, like random access.

That said, there's no reason why one could not make such a comparison. Indeed, we do include one comparison with JPEG2000 on the zfp homepage based on the implementation of Woodring et al. for compressing climate data using JPEG2000.

I also do work closely with climate scientists at NCAR who favor zfp and fpzip over GRIB2. See this paper and references therein on further discussion of and comparisons with GRIB2.

Finally, the obvious "hybrid" would be to apply zfp to quantized GRIB integer data. However, zfp is not all that well suited for lossless compression (fpzip performs far better). Note that the first stage of zfp compression (the block-floating-point transform) essentially performs uniform quantization (in fixed-accuracy mode). In conjunction with the other zfp steps, this quantization allows bounding the error and exploiting higher-dimensional correlations.

A perhaps better compromise is to use zfp with HDF5 via the H5Z-ZFP plugin.