add optional support for zstandard compression

[edwardhartnett]

I suggest we fully support the ZStandard library in the netCDF C, Fortran, and Java libraries. (This suggestion is part of a larger effort to improve netCDF compression #1545).

Compression is Essential, and Zlib is Slow

Compression has become a core component of netCDF. Large data producers, like NOAA, NASA, and ESA, all must compress their data in order to fit into their storage systems, which were sized with the assumption that data compression would be used operationally. That is, none of these data providers have the disk space to process and store these data uncompressed. Compression is essential.

The slowness of zlib with today's giant data files has become a real problem for ESA with the Copernicus satellite program, which has been generating large amounts of netCDF-4 data for many years. As with NASA Earth observing missions, the amazing increase in the capabilities of space instruments has been accompanied with a dramatic increase in the size of the data - the slow speed of zlib is becoming a serious problem.

As documented in #1545 NOAA is also running into the limits of zlib. For the recent GFS release, we dodged the bullet by adding the ability to use zlib with parallel I/O to netcdf-c-4.7.4. This allowed NOAA to regain its lost time budget and write the forecast data on time. But future versions of the GFS are planned with higher resolution. As we all understand, doubling the resolution in time and space results in a 16-fold multiplication of the data sizes, and, consequently, the time required to compress the data.

Why ZStandard?

The Zstandard compression library, currently available to netCDF users through the CCR project, provides significant performance improvements over zlib.

This chart from our recent AGU extended abstract shows typical performance improvement achieved with ZStandard:

When used with the new quantize feature, Zstandard performance is even more impressive:

Zstandard also offers a much wider range of tradeoff between compression and speed. For example in Klower's paper a setting of -10 was used to get very fast compression, which only compressed the data 50%. This allows operational requirements to be met. Zlib does not offer this range of speed/compression trade off. It is not possible to make zlib go much faster, whatever setting is used.

ZStandard is free and open software, readily available on all platforms: Unix, MacOS, and Windows. A reference C library is available which can be compiled anywhere, and pre-build Zstandard libraries are available on most package management systems. A pure-Java implementation is also available from the Zstandard website.

The Zstandard web site displays a table of Zstandard performance compared to other compression libraries, including zlib. They show better performance than I have measured but similar. They show a 5x speedup for writing, 4x for reading. (I have not measured read time - NOAA is more concerned with write times.)

Implementation

How would this look in implementation?

C and Fortran

The C and Fortran implementations are available in the CCR. They are very similar to the existing functions which currently support zlib:

    int nc_def_var_zstandard(int ncid, int varid, int level);
    int nc_inq_var_zstandard(int ncid, int varid, int *zstandardp, int *levelp);

In Fortran:

  function nf90_def_var_zstandard(ncid, varid, level) result(status)
  function nf90_inq_var_zstandard(ncid, varid, zstandardp, levelp) result(status)

The build systems would be updated to optionally include Zstandard, as Szip is handled now. Extra items in the build summary and netcdf_meta.h will help end users know if Zstandard is supported in any particular build. The Zstandard filter code would be included with netcdf-c, and installed in the HDF5 filter directory for the user.

(If released in the netCDF C and Fortran libraries, these functions would be removed from CCR.)

NetCDF-Java

There is a pure-Java Zstandard library. Code to use it as a filter would have to be written by the Unidata netCDF-Java team.

My understanding is that only Java read code would have to be modified. The pure Java part of netCDF-Java only reads NetCDF/HDF5 files. In this case the use of Zstandard would be similar to the way zlib is handled.

The NetCDF-Java library would be able to use the C library to write netCDF/HDF5 files with Zstandard compression, but support would have to be added to call the new functions.

Conclusion

The inclusion of Zstandard in all netCDF libraries will be a significant improvement for large data producers such as NOAA, NASA, and ESA. Zstandard provides both better performance and a much better range of performance, allowing data producers to tune the compression vs. speed of their data to match sometimes stringent operational requirements.

All readers of compressed data will also benefit from a performance speedup with Zstandard.

The greater ease of handling larger and larger data sets will also bring benefit to the science community, as more data can be written and distributed within the operational constraints of large data producers.

Although these benefits are currently available if all concerned install CCR, I believe there will be significant benefit to the netCDF community if Zstandard were supported out of the box by all netCDF libraries.

@DennisHeimbigner @WardF @dopplershift @lesserwhirls @czender and @gsjaardema your input is very much requested...

References

CCR web site: https://github.com/ccr/ccr

Zstandard web site: http://facebook.github.io/zstd/

Compressing atmospheric data into its real information content
Milan Klöwer, Miha Razinger, Juan J. Dominguez, Peter D. Düben & Tim N. Palmer, https://www.nature.com/articles/s43588-021-00156-2

Quantization and Next-Generation Zlib Compression for Fully Backward-Compatible, Faster, and More Effective Data Compression in NetCDF Files, Hartnett, Zender, Fisher, Heimbigner, Hang, Gerheiser, Curtis: https://www.researchgate.net/publication/357001251_Quantization_and_Next-Generation_Zlib_Compression_for_Fully_Backward-Compatible_Faster_and_More_Effective_Data_Compression_in_NetCDF_Files

[DennisHeimbigner]

Query: is zstandard the same as zlib-ng?
Also, is the zstandard API different from zlib (that is, is it a direct replacement).?

[czender]

@DennisHeimbigner Zstandard is distinct from zlib-ng, which is a drop-in replacement for zlib. The Zstandard API is different. You can see the Zstandard C API in action in this CCR codec code: https://github.com/ccr/ccr/blob/master/hdf5_plugins/ZSTANDARD/src/H5Zzstandard.c.
I would underscore @edwardhartnett's point above that Zstandard has wide support in a variety of native languages including Java.

[DennisHeimbigner]

Got it. What do you use as the zstandard implementation you use (i.e. code base)?

[edwardhartnett]

@DennisHeimbigner the zstandard code can be found here: http://facebook.github.io/zstd/

However I installed it on ubuntu with:

sudo apt-get install zstd

[edwardhartnett]

@DennisHeimbigner as noted by Charlie, Zstandard is a completely different library from zlib. It has it's own filter number in the HDF5 ecosystem.

zlib-ng is a great instant upgrade for everyone. It causes no compatibility issues and requires no code changes, it's fully backward compatible with old installs of netCDF, and it can be done today. But it only takes us a few steps back from the rising tide of data.

Zstandard offers features beyond zlib/zlib-ng, including better performance and compression. Fully supporting it in netCDF will take some time and effort. Once available it will not be backward-compatible with old installations of netCDF - readers will have to upgrade to the latest netCDF release. So it will take time to become widely useful.

But once implemented and digested by the community, Zstandard will allow our users to take a few more big steps back from that rising tide. ;-)

[DennisHeimbigner]

I see there is also a numcodecs std for this, so adding some tests for it should
be pretty easy.
But at some point, I would to consider leaving this in CCR and adding the necessary
codecs libraries to CCR.

[dopplershift]

To me the zstd algorithm definitely looks promising. If we were to do this, my only concerns would be:

1. Making sure the netCDF standard ("NUG") is properly updated first
2. Planning for this work in the netCDF-C, netCDF-FORTRAN, and especially netCDF-java roadmaps/schedules so that this is available in both libraries before we go shouting it from the rooftops.

[dopplershift]

cc @haileyajohnson

[edwardhartnett]

I am happy to update the NUG, if necessary.

The code for C/Fortran is already written, documented, and released in CCR. So it can be moved into the netcdf-c and netcdf-fortran libraries in a matter of hours, if the consensus is that zstandard should be fully supported in netCDF. I can do this at any time.

The Java library is the open question.

[edwardhartnett]

@DennisHeimbigner I don't understand your comments about codecs and testing.

A few other stray points that have not yet been mentioned:

• In NetCDF needs better and faster compression, especially for HPC applications #1545 I suggested adding a handful of new compression libraries. I think that if ZStandard is added, there will be relatively little appetite or need to add more. The others do not seem to offer any dramatic benefits over ZStandard. So I think adding ZStandard may be enough to satisfy compression needs for some years to come.
• I note that ZStandard is also used in Zarr. No doubt it is preferred there, for the same reasons we would like to use it in netCDF/HDF5 files.

[DennisHeimbigner]

The codec stuff is necessary for nczarr filter support.
Except for testing, I do not see what it means to add zstandard support
to netcdf-c. It is as simple/hard as adding any other filter.

[edwardhartnett]

To add zstandard support means:

• Look for zstandard library at build time in configure/cmake.
• Include additional functions nc_def_var_zstandard/nc_inq_var_zstandard.
• If zstandard found, build and install the zstandard filter as part of netcdf-c build.

This would make zstandard a privileged filter - one with greater support within the netCDF C, Fortran, and Java libraries.

As noted, zstandard is already fully supported in CCR. The point of this issue is to move that support into the netCDF C and Fortran libraries, and to get a plan for adding it to netCDF-Java.

[rsignell-usgs]

The default compressor in xarray when writing zarr is actually lz4:

 'compressor': Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0),

which worked even better than zstd in a genotype benchmark described in a blog post by @alimanfoo.

The conclusions of the blog post were:

Conclusions

For maximum all-round speed, Blosc with LZ4 and no shuffle is ludicrous, compressing at 11G/s and decompressing at over 7G/s.
For higher compression ratios, Blosc with Zstandard is excellent. Adding the bit-shuffle filter increases compression ratio even further to over 50X at a moderate cost to decompression speed.
Using Blosc with multiple threads accelerates both compression and decompression, but larger (16M) chunk sizes were needed to see the benefits (see also below).

Naturally there were benchmark result caveats also:

Caveats

There are multiple interacting factors at work here, including data, software and hardware. Your mileage may vary substantially. Running your own benchmark is highly recommended.

It would be great to have support for zstd for sure though.

[DennisHeimbigner]

I was afraid of this. Everytime we get a new and "better" compressor, it becomes
desirable to build it into netcdf-c. The only reason we support zlib and szip
is because they are in HDF5.
I really think a better idea is search at build time for ccr and let it do all the work.

[edwardhartnett]

The reason to put zstandard into netCDF is that it makes it easy for users to converge on something better than zlib, without splintering the community into many different compressions, each unreadable by other users until they figure out how to locate, build, and install the correct filter.

My understanding was that there was some concern at Unidata about this splintering of the community.

The current state of play is that users can use CCR and have Zstandard and soon BLOSC. NetCDF-Java can catch up as best it can, or be left behind. This is the path we are now on.

The goal with this issue is to guide the community into an acceptable, common choice. By providing full support for Zstandard, we make it the most attractive of the compression options. (BTW the LZ4 filter in HDF5 does not work, so that's why CCR does not support LZ4.)

What I would like to tell the GFS team is that netCDF will soon support a better compression method, which will help meet their operational requirements. If not that, then NOAA can use CCR and Zstandard. However, Unidata can then expect to get many support requests from users trying to read NOAA data, and needing to have filters and CCR explained to them. ;-)

And if ESA goes with BLOSC, NASA with LZ4, and NOAA with Zstandard, then there will be more work for the netCDF-Java team, or else they will simply have to give up on being able to read all netCDF files.

If there is concern for splintering of the community, we should select a common, acceptable path forward.

@DennisHeimbigner the amount of work to give zstandard preferential treatment is trivial. We spend a lot more time discussing it than it would take to do it...

[jswhit]

Is there a way to replace zstandard with zlib in a backwards compatible way? Perhaps the lib could detect the compression algorithm at runtime, and decompress accordingly. Zstandard could be made the default compression algorithm going forward?

[WardF]

I would be hesitant to replace zlib altogether unless Zstandard were as ubiquitous across platforms and installations. Making it the default, when available, would be in line with the recent changes to our XML handling (if libxml is available, use that, if not, fall back to the bundled XML parser).

[edwardhartnett]

No there is no way to replace zlib with zstandard. That would do a lot of violence to the way filters are handled in netCDF.

Zlib is there and will always remain. Zstandard would add a new alternative.

No form of compression is on by default - the user must always select a compression, if it is desired.

[DennisHeimbigner]

BTW the LZ4 filter in HDF5 does not work, so that's why CCR does not support LZ4.)

As an aside. If memory serves, libblosc contains LZ4 impl. why not use that since you are already
adding blosc.

[haileyajohnson]

We've been working on a better "plug and play" way of handling filters/compressors in the netCDF-Java library. Right now, if there's already a Java implementation of zstd, it would be trivial to add it to the package. The current state is that zlib, shuffle, scaleoffset, and several 32-bit checksums are implemented in the core package, and users can supply others as plugins.

But I'm also interested in working on a way to lean on CCR, in large part so that the java library can more easily stay in step with the others. Whatever we do, the java interface to the C library will likely need some work.

As a side note, there's definitely a lot of interest from the Zarr community on this topic :)

[DennisHeimbigner]

Sorry, but I just don't buy this "compressor du jour" approach to things.
Today it is zstandard tomorrow it is QOI.

[DennisHeimbigner]

In thinking about this, it looks like we need to provide guidelines about
what goes into ccr and what goes into netcdf-c. Also, who is currently
responsible for maintaining ccr? Ed or Charlie or someone else.

[czender]

Ed and I are jointly developing and maintaining the CCR. Except that he's a wizard and I'm a muggle at all the infrastructural and autotools stuff, which therefore falls to him.

[edwardhartnett]

@DennisHeimbigner I find your comment about compression du jour to be a bit odd.

We have been using zlib (and szip) for 15 years. Seems like an update of compression methods once every 15 years is not too frequent.

When I meet netCDF users at meetings or conferences, I always ask what netCDF could be doing better to meet their science needs. The answer, invariably for the last five years (and longer), has been: "improve compression." As a member of the NOAA GFS team, I now also see this need from within NOAA.

There seems to be a lack of appreciation at Unidata for the difficult decisions that large data producers must make. We want them to use netCDF for better interoperability and transparency, and they want that too, but they cannot sacrifice their operational requirements. If netCDF does not provide better compression, then data producers will start to find other solutions. Even now, unwelcome compromises about what can be written must be made. With higher resolutions here at NOAA, we will once again be over our I/O time budget in the GFS. What will happen then?

The current course we are on is that users like NOAA will add CCR to get better compression methods. This will lead to fragmentation and extra work for the netCDF-Java team, as they attempt to cope with multiple compression filters. Widows users may find data inaccessible, if it has been compressed with software not ported to Windows.

Thus my suggestion that one new, good compression method be added to netCDF, to try and head off this fragmentation.

[DennisHeimbigner]

As I mentioned, the reason for zlib and szip support was because they were directly supported
in HDF5.
In principle, I have no particular objection to defining a --enable-zstandard option
plus nc_def_var_zstandard.
But it would seem to me that adding --enable-blosc and nc_def_var_blosc would make
more sense since that covers zstandard as well as LZ4 and a couple of others.
But this seems to be seriously encroaching on the use cases for CCR, and over time would
obviate the need for CCR.

[edwardhartnett]

Your point abut blosc is a good one, but then it makes the netCDF Java team's work harder - they will have to cope with multiple decompression filters instead of one...

[WardF]

We met on this yesterday and I'm drafting a big old chunk of text, but the TLDR version is we are interested in incorporating this, but we have a couple of questions. Looking forward to seeing this inclusion. I didn't want to head off into the weekend on a note of radio silence, so I just thought I'd mention that. Stand by for the larger post.

[edwardhartnett]

OK, great. I will take a look at blosc and see if we can get it working in CCR so we can all take a look at it in action.

I think this will take some time for us all to reach consensus, and for the work to be done. So I don't think this is going to happen before the next release or any time in the immediate future. ;-) Hopefully sometime in the first half of this year, if we can all agree we've found a good solution.

Meanwhile, I'm very excited to see the new quantize feature released, in both C and Fortran. Used with the existing zlib compression, this will yield much better and somewhat faster compression for all uses. NOAA is waiting for this release, with the plan to then use the quantize feature in most (or all) of the models that run in the UFS, taking advantage of the smaller data sizes it will bring, which mean less time spent reading, writing, and transmitting data. So the quantize feature is eagerly awaited and will bring immediate benefit to large data producers.

[jswhit]

@WardF still waiting breathlessly for the larger post you promised....

[WardF]

@jswhit Thank you for the bump, it is appreciated :). This didn't fall off my radar, it just ended up sitting below the surface, covered by other things we're having to address at Unidata.

Ok! As previously mentioned, we think we should move forward with this, and I'm excited to see what Zstd support in the core netCDF-C library can potentially accomplish. In support of @edwardhartnett's comment, I'll open up a discussion over at the netCDF-C Github discussion page for us to hash out the details of the implementation.

Reading back through the thread, we (the Unidata team) wanted to lay out the position we're in and highlight some of the considerations we are obliged to think about beyond the 'how easy will this be to add'. Not that that isn't also a consideration, and when there's an offer to add it via PR, all the better ;).

In terms to adding bespoke functionality to the core netCDF-C library, we have to have a deliberate consideration of the following:

• What is the technical debt incurred and what happens if any new dependencies used 'go away', dry up, stop being maintained, etc. We can't just abandon the files written using these features, so what would supporting a read-only version look like?
• What is the impact on other netCDF projects? The Fortran/C++/Python interfaces? The netCDF-Java & THREDDS projects? It wouldn't be a great look to have a lot of unreadable files start showing up in a particular ecosystem (THREDDS would be the nightmare scenario, I think) because the appropriate developers weren't consulted and kept in the loop.
• Does the proposed change server the community at large, or is it a change that only a very specific group of users would benefit from? Frankly, we get a lot of requests for features that a particular person or team would like but provide little-to-no utility in terms of the package-at-large.

These questions are considered before we even get to the point of considering technical details (what does the API look like, are the packages available across platforms, are they available through package management systems or require installation from source, etc). As a result of these considerations, we often have to move more slowly than any of us would like when making these decisions; even when the answers seem clearcut, we have to consider the questions :).

That said, the internal discussion more or less concluded that Zstd isn't going to dissapear, the risk of assuming an untenable technical debt is very low, the potential benefit to the broader community is high, and the workload of implementation across projects will not be crushing. Hooray, +1, lets discuss the details here.

Thanks all for your patience!

[DennisHeimbigner]

int nc_inq_var_zstandard(int ncid, int varid, int *zstandardp, int *levelp);

Ed, a question about this. This signature uses a zstandardp flag to indicate
if the variable uses zstandard compression.
I would prefer an alternative of returnin NC_ENOFILTER if zstandard is not used. This
reduces the the function to this form:

int nc_inq_var_zstandard(int ncid, int varid, int *levelp);

Comment?

[edhartnett]

I don't think it's a good idea to cause an inq function to error, if the user has not done anything wrong. Currently, no inq function will give an error as a way of communicating results, only for errors. Also, many Fortran users do not check return codes. So it would be best if this worked similarly to other inq functions.

…

On Fri, Feb 11, 2022 at 3:09 PM Dennis Heimbigner ***@***.***> wrote: int nc_inq_var_zstandard(int ncid, int varid, int *zstandardp, int *levelp); Ed, a question about this. This signature uses a zstandardp flag to indicate if the variable uses zstandard compression. I would prefer an alternative of returnin NC_ENOFILTER if zstandard is not used. This reduces the the function to this form: int nc_inq_var_zstandard(int ncid, int varid, int *levelp); Comment? — Reply to this email directly, view it on GitHub <#2173 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCSXXG2OIZWIRAQWFZ6SOTU2WCJXANCNFSM5LCTA5NA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[edwardhartnett]

OK, this has been done, although see #2294 for trouble in paradise.

I will close this issue however.