Allow users to turn on szip compression for variables in netCDF-4 files

[edwardhartnett]

HDF5 offers native access to zlib and szlib compression. However, at the time I wrote netCDF-4, szlib had an unclear license. So Russ and I decided to make it a read-only capability in netCDF-4. That is, you can read HDF5 files written with szlib, but you could not create them.

(Probably we were being too cautious anyway. HDF5 uses it and no one cares.)

The szip compression library is beloved by NASA Goddard. It does better then zlib with arrays of floating point data.

Anyway, Elena has explained that they have changed their license and are now fully free software. So we can allow users to turn this form of compression on, pretty easily. (I actually used to have a function to do this, but I took it out when we got concerned over licensing.)

This is part of #1545

[ArchangeGabriel]

Interesting, didn’t now that they turned to free software licensing (can you give a link for this though, I still read “free for non-commercial use”?). We were relying on libaec to provide szip support on Arch, maybe we should re-evaluate that then.

[epourmal]

There is SZIP implementation from German Climate Computing Center that has BSD type license. It is fully compatible with SZIP supported by The HDF Group. You can get the source from here.

[ArchangeGabriel]

@epourmal Yeah, just as I implied above, this was the de-facto replacement when SZIP was non-free. But if they are now free, re-evaluating SZIP vs libaec might be interesting.

[edwardhartnett]

So here's the question: how do we handle it when people do not have libaec installed? Some alternatives include:
1 - Fail configure and demand they install it or build with --disable-netcdf-4.
2 - Build, but only include nc_def_var_szip() if libaec is installed?
3 - Include libaec in the builds (both autoconf and cmake), and build and install it if not already present.

3 seems safest - all users are guaranteed to be able to read files compressed with libaec.

[epourmal]

I agree, 3 is the safest. You can also include the source of szip with netCDF-4 source code.

[epourmal]

Forgot to mention that in HDF5 szip is a required filter, i.e., H5Dwrite will fail if the filter is not found for some reason. Do you build HDF5 libraries when you build netCDF-4 or some preinstalled binaries are used (probably both)? You will need to check that HDF5 was built with szip.

[edwardhartnett]

OK, so szip cannot be installed after HDF5 is built, however, other filters can?

To answer your question, we use an installed HDF5 library. We don't know if the user just built it, or installed it with apt-get, or what. We know nothing except what we can check. ;-)

[dopplershift]

Does libaec build on windows at all? Conda-forge has windows builds for libaec disabled. I have severe reservations of making this a major feature in netCDF if a major client platform won't be able to use it.

[dopplershift]

Actually, I'm going to take that a step further: I have HUGE problems with making it possible to use the netcdf-c library to write files that you can't open on Windows.

[edwardhartnett]

@dopplershift this horse has already left the barn. ;-)

With the recent addition of filters, netcdf-c can use any HDF5 filter, which means users can already write netcdf files with any filter. So it is quite possible to write netcdf-4 files which cannot be used on Windows, if there are filters that can't build on windows (and there may be).

I agree that full windows compatibility is a requirement for the compression features I am adding to netcdf-c. I will add that all compression features must be available in the F77 and F90 APIs as well. The Fortran APIs are how all modeling and climate groups use netcdf to write the giant data sets that are most concerned with compression.

My goal in the current effort is to add compression that can be assured on all platforms. That is, I want to not just use some filters, but actually ensure they are packaged with the netCDF distributions (automake and cmake) so that all users are known to have them. (If we find them already installed, we do nothing. If missing, we install them when netCDF is installed.)

This includes Windows. So we should select compression filters that work well, that can build under autotools and cmake, and that work on Windows. Any compression filters that don't meet those criteria will probably not make the cut for what I am trying to accomplish with this compression work.

But not netcdf-java. I rather doubt that all the filters exist in Java. So netcdf-java may have to fall back on using the C library for reading, as well as writing HDF5 some files. This is inevitable in any case. Eventually we will start using some new HDF5 features which John Caron did not know about, and so did not code into netcdf-java.

However, we must also not let the widespread penetration of netcdf hold us back from improving it.

As with the releases of netcdf-4.0 and netcdf-3.0, we must honor our commitments for full and complete backward compatibility in API and data formats. All existing netCDF code should continue to work, and all existing netCDF files continue to be fully readable by all future versions of the library.

[dopplershift]

With the recent addition of filters, netcdf-c can use any HDF5 filter, which means users can already write netcdf files with any filter.

I'm aware of this. I'm pushing back against adding more APIs to the C library that simplify the process of creating such files and exacerbates this problem. Weren't you just arguing how hard it is to use the filter API so that few people do it and that's why we need the simple API?

However, we must also not let the widespread penetration of netcdf hold us back from improving it.

That's an easy statement to make when you're not the one fielding support requests over on the netcdf-java repo. I'm all for making improvements to better serve our community. But such decisions need to be made in regards to what's good for the netCDF data model, file format(s), and above all our community. netcdf-c is but one implementation that Unidata maintains, and additions and changes to on-disk format need to be made considering the entire portfolio of netCDF implementations. Confusion when one version of the library can read a netCDF file and one can't is NOT good for the community.

So netcdf-java may have to fall back on using the C library for reading

Again, easy to say since you're not responsible for handling the support requests for clients of netcdf-java and having to ship Java packages that now would REQUIRE shipping multiple platforms' worth of compiled libraries. It's one thing to put some of that burden on people running the TDS (and even then just to write a certain format)--it's a whole other when that means we need to worry about bundling this with every copy of IDV, Panoply, or any other user of the netCDF-java library.

I love aspirational ideas and we should be considering every transformative change we can make to better serve the needs of the community. I think you have this covered really well right now @edwardhartnett; so I'm going to be here to represent reality. And the reality is, we support both netCDF-java and netCDF-c and we need to consider both, in terms of support risk and technical effort, when we're talking about new features that impact how data files are written and what clients will be able to read them. We're already doing that with the zarr work, and work on compression should be no different.

[DennisHeimbigner]

At least twice now, we (Ward and I) have rejected the idea
of netcdf-c including a bunch of additional filter implementations
because of security and maintenance issues. I for one
still stand by that position.

One partial approach would be to have a new include file,
netcdf_known_filters.h say. This file would contain wrappers
for nc_def_var_filter for specific filter s -- bzip, etc.
These wrappers would accept filter specific arguments and translate them
to proper form for nc_def_var_filter.
This would allow users to use a given filter with more semantically
specific parameters.
Notes:

1. this file would not be part of netcdf.h, so we do not pollute
   the core API.
2. Users could propose additions to this set of wrappers.
3. Presence of a wrapper in netcdf_known_filters.h would not
   mean that the implementation of that filter is available.
   The user is still responsible for finding and installing an
   implementation.
   None of this does addresses the problem of having files
   that cannot be completely read because it uses missing filters.
   We could build a program that takes a .nc file and prints
   out the filter ids of any filters used by that file (and which variables use them).
   Remember that until libaec, we had this problem already
   with the szip filter because it was supported, but no one had
   a non-proprietary implementation.

[edwardhartnett]

Well this has been a very interesting discussion!

@dopplershift do you object to Dennis' proposal?

@DennisHeimbigner would you be willing to allow me to extend this idea to the Fortran APIs?

I just participated in a NOAA telecom. Zlib is not meeting our operational constraints and we are now looking at other filters, starting with libaec/szlib. Speed of reading is a concern, because zlib is slow to uncompress data. Parallel I/O helps with that on HPC, but many programs run sequentially, or on non-HPC systems.

[DennisHeimbigner]

Having Fortran follow C is fine I am sure.

[DennisHeimbigner]

One other thing. Is there any agreement in the community about
in picking a replacement for zlib?

[dopplershift]

@edwardhartnett I think Dennis' proposal is incomplete because:

None of this does addresses the problem of having files
that cannot be completely read because it uses missing filters.

I'm not supporting any concept that leads to a proliferation of files that only netcdf-c can read. You're talking about adding features for the explicit purpose of allowing one of the major data creators in our field to create a new variant of files (which I fully support). What I'm trying to say is, any path we decide upon MUST, in no uncertain terms, include a plan that the netcdf-java team is comfortable with (cc @lesserwhirls). Full Stop. I'm not sure I can make that requirement any clearer.

[dopplershift]

To put another way, at this point I think it's best that the netCDF team, which includes both netcdf-c and netcdf-java developers, figure out the best way to proceed here.

[epourmal]

One other thing. Is there any agreement in the community about
in picking a replacement for zlib?

We experimented with zlib and some VIIRS NPP files. HDF5 shuffling filter and zlib gave us a pretty good compression ratio and encoding/decoding speed on integer data.

It is probably a not a good idea to replace zlib. Users should have access to variety of compression methods since there is no "universal" compression solution and goals are different; we cannot know if someone wants to minimize storage space, or minimize time for encoding or decoding, or encoding only, or decoding only, or ...?

In our experiments we were trying to optimize for file sizes, and we got the best result when we appled different compression methods to different datasets (szip, shuffle+gzip, and no compression for some datasets because of HDF5 overhead).

[DennisHeimbigner]

I did not mean to imply that we obsolete zlib. Rather if the community
settles on a single compression that had wide-spread use, wemight
think about supporting that compressor in netcdf-c.

[DennisHeimbigner]

WRT Java. My summary of the issue is that for new filters, we either

1. force use of JNA or
2. make an attempt to modify the Java HDF5 implementation
   to support additional filters.
   Are there other approaches?

[WardF]

To put another way, at this point I think it's best that the netCDF team, which includes both netcdf-c and netcdf-java developers, figure out the best way to proceed here.

We can discuss this at the next netCDF team meeting; we'll schedule when we can include @lesserwhirls as well. One of the issues that was raised with the plugin functionality was that end users would now have the ability to create data that is not broadly readable. This was a trade-off between flexibility for end users, and broad interoperability via the plugin/filter functionality. I agree with the sentiment expressed in this thread, that the core functionality provided by libnetcdf emphasize cross-platform compatibility.

[lesserwhirls]

WRT Java. My summary of the issue is that for new filters, we either

1. force use of JNA or
2. make an attempt to modify the Java HDF5 implementation
   to support additional filters.
   Are there other approaches?

I think this has it in terms of options for reading. In my ideal word, we would not need to force the use of JNA for reading any blob of bits called netcdf, ever. Even better would be that any blob of bits called netcdf would be fully reproducible, read and write, with more than one implementation and no single point of failure. I say that knowing that it's, perhaps, controversial, and only my opinion. For writing, we're kind of stuck with JNA, at least for using HDF5 as the persistence format for the netCDF enhanced data model. I don't anticipate this will be the case with, say, using Zarr for the same purpose (reading or writing).

The netCDF-Java HDF5 code already has support for a few filters, and adding to, or making those pluggable as a service provider, would not be complicated. The big question would be if a suitable compression library exists on the java side. Ideally, any compression schemes that would go into the core C library would produce data that could be readable by netCDF-java and other libraries, and vice versa...also (and stronger than ideally), available freely at no cost, and with minimal restrictions (can't get around ITAR, for example). Certainly we could write out compressed data using jpeg 2000 on the java side and call it netcdf, but we don't.

If we take the route having a canonical list of filters (which I think should be very clearly defined for netCDF-4, and not simply a reference to whatever the version of HDF you are using supports), I'd strongly recommend that be done in a more visible way than a header file in this repository. Certainly some sort of header file describing the filters and parameters would be needed at the C level, but I think that's a C implementation detail. I would say proposals for the addition of a new filter would not come in the form of a PR against netcdf_known_filters.h.

[edwardhartnett]

Please discuss among yourselves and let me know how you would like to proceed. ;-)

[czender]

As Ward and Dennis know (because they supported it), I submitted a proposal to the NSF CSSI program aimed at addressing some of these issues. A similar proposal that I submitted last year was rejected, and we'll have to wait until June-September to see if this new one is funded. The basic idea is a Community Codec Repository (CCR) where netCDF filter code lives, and that can be enabled to be built/installed on the user's machine during netCDF build-time (with --enable-ccr) or installed as its own package independently. This minimizes risk/maintenance for Unidata, and is intended that users can count on interoperability with any CCR codec with minimal-to-no effort. The proposal in full is here.

I agree with Ed and others that netCDF needs more modern (faster/better) lossless compression, and lossy compression as well. Lack of this hinders climate/weather modeling/research (and causes needless GHG emissions to store noise). Today I obtained the GitHub site http://github.com/ccr to host the CCR. If you are interested, please join the nascent efforts to architect the CCR so that it meets your project's needs. Feedback and code contributions are welcome there.

[epourmal]

It would be good to coordinate this effort with The HDF Group.

We do have our Codec Repository for regression testing with HDF5 and we do provide binaries for the codecs with HDF5 binaries (see our download pages). The repository is open and can be moved to a better place.

Let's coordinate and cooperate :-)

[czender]

@epourmal I get "Invalid username or password." when I use my Atlassian login on that codec repository. How do I gain access?

[epourmal]

Please try again; we fixed permissions

[czender]

Thanks, I have access now. I agree that we should coordinate and co-operate, because the interoperability of compressed datasets is of paramount importance to users and therefore adoption of non-DEFLATE algorithms. That is why Aleksandar Jelenak (@ajelenak-thg) of the HDF Group agreed to be a collaborator (like Ward and Dennis) on my submitted proposal to NSF. As you know, A. currently co-chairs NASA's Dataset Interoperability Working Group.

[epourmal]

Thank you, Charlie! I guess THG needs a better collaboration internally too :-)
If you at Winter ESIP, let's meet and discuss.

[edwardhartnett]

OK, this has been a great discussion, about the general questions of newer filters for netCDF.

But let's set aside talk of all the other compression methods, and return to the specific topic of this issue: do we reinstate the nc_def_var_szip() function, to match nc_inq_var_szip(), which is already there?

The question of support for other, newer, compression filters is separate and may be continued elsewhere (for example in #1545), but the current issue is just about szip, which is currently partially supported by netCDF.

From netcdf.h:

/* Find out szip settings of a var. */
EXTERNL int
nc_inq_var_szip(int ncid, int varid, int *options_maskp, int *pixels_per_blockp);

Shall I submit the PR re-instating the nc_def_var_szip() function?

[DennisHeimbigner]

I would say yes now that libaec is available.

[edwardhartnett]

OK, it turns out that HDF5 (as with zlib) offers special built-in support for szip.

What this means is we can either use szip with the filter API, or use the built in HDF5 szip functions. Both will work.

BUT! The filter method will only work when shared libraries are being built. NOAA HPC sysadmins have a (perhaps unreasonable, perhaps not) aversion to shared library builds. The NOAA GFS is built all static. Other HPC users of netCDF have told me the same.

So in this case, we want to use the built-in HDF5 functions. By doing so, we get static build functionality for free, whereas with the filter API we have to take extra steps to get static builds.

[epourmal]

Ed, One doesn’t need shared szip. Shared libraries are needed for dynamically loaded filters. If you are going to use specific filters at NOAA you can always built statically even with the filters that are not supported directly by HDF5. Let’s talk off-line. Elena

…

Sent from my iPhone On Jan 3, 2020, at 8:20 AM, Edward Hartnett <notifications@github.com<mailto:notifications@github.com>> wrote: OK, it turns out that HDF5 (as with zlib) offers special built-in support for szip. What this means is we can either use szip with the filter API, or use the built in HDF5 szip functions. Both will work. BUT! The filter method will only work when shared libraries are being built. NOAA HPC sysadmins have a (perhaps unreasonable, perhaps not) aversion to shared library builds. The NOAA GFS is built all static. Other HPC users of netCDF have told me the same. So in this case, we want to use the built-in HDF5 functions. By doing so, we get static build functionality for free, whereas with the filter API we have to take extra steps to get static builds. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1546?email_source=notifications&email_token=ADLFT3O6VSRANIAWKHMFYNLQ35CQZA5CNFSM4JQH2IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIBG5LY#issuecomment-570584751>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADLFT3M7OJXIS2QD76RMPLTQ35CQZANCNFSM4JQH2IKA>.

[edwardhartnett]

OK, turns out this is already in place, and uses H5Pset_szip(), just as we want. The following code is in nc4hdf5.c:

    /* If the user wants to deflate the data, set that up now. */
    if (var->deflate) {
        if (H5Pset_deflate(plistid, var->deflate_level) < 0)
            BAIL(NC_EHDFERR);
    } else if(var->filterid) {
        /* Handle szip case here */
        if(var->filterid == H5Z_FILTER_SZIP) {
            int options_mask;
            int bits_per_pixel;
            if(var->nparams != 2)
                BAIL(NC_EFILTER);
            options_mask = (int)var->params[0];
            bits_per_pixel = (int)var->params[1];
            if(H5Pset_szip(plistid, options_mask, bits_per_pixel) < 0)
                BAIL(NC_EFILTER);
        } else {
            herr_t code = H5Pset_filter(plistid, var->filterid, H5Z_FLAG_MANDATORY, var->nparams, var->params);
            if(code < 0) {
                BAIL(NC_EFILTER);
            }
        }
    }

Looks like @DennisHeimbigner put this in as part of the filter work. So that makes this issue easier, I will just add the nc_def_var_szip() function back in...

[dopplershift]

@DennisHeimbigner can you look to see what options we have to implement this in netcdf-java?

[DennisHeimbigner]

If you are willing to include the filter code as part of your client
application, then there are 3 functions in include/etcdf_filter.h
that allow you to specify that filter to be used for nc_def_var_filter.
See Appendix B of tje file docs/filters.md.
This should work with static builds.

These functions are:
EXTERNL int nc_filter_register(NC_FILTER_INFO* filter_info);
EXTERNL int nc_filter_unregister(int format, int id);
EXTERNL int nc_filter_inq(int format, int id, NC_FILTER_INFO* filter_info);

[DennisHeimbigner]

It occurs to me that alternatively, one could compile the filter
and add it to the static library archive file (libnetcdf.a)

[Dave-Allured]

Rather than bundling, I suggest deferring all management of filter libraries to HDF5 and user responsibility. Currently netcdf-C contains no direct dependencies on any filter libraries, not even zlib. I think this would be the easiest and simplest way forward for szip as well as arbitrary filters.

If a user has the necessary libraries installed, then their installed netcdf can read the related compressed files. If a certain library is missing, then they get a meaningful error message at the appropriate time. Then they have the usual remedy, install and/or update their libraries. This should work the same way in principle for both static and shared builds.

Certainly Unidata should continue to publish a recommended set of libraries to support the preferred set of format variations and filter options.

[DennisHeimbigner]

I agree that it is preferable to leave it to HDF5.
But as it stands, this statement is false

This should work the same way in principle for both static and shared builds.

because there is no way fir HDF5 to find and load the filter code
statically that I am aware of.

[edwardhartnett]

@DennisHeimbigner your points are valid for other filters, but none of this is necessary for the szip filter (the topic of this issue) - it already ships with HDF5, so is always present. (Also there is a way to build the filter code statically, but I have not yet figured it out or tried it.)

What you suggest (that netcdf-c contain the filter code) has merit and I did originally support that idea. But then, working with filters for a bit, and reading concerns from you, @WardF and @dopplershift, I realized that HDF5 should actually fill this role. The filter code is much closer to HDF5 code than it is to netCDF code.

To take the LZ4 filter as an example, it is only one C file. It would be trivial to include it in every HDF5 build, and (just as with zlib and szip) check for it during configure, and built the filter if liblz4 is found.

I am going to suggest this to the HDF5 team (howdy @epourmal!) Of course this means I will volunteer to submit changes to their build systems to support this - which will be easy. If that can be achieved, then netcdf-c faces a situation where all HDF5 installs can be required to have LZ4 (like zlib) or just support it if it is present (like szip).

@Dave-Allured, what you propose is essentially the current situation. Continuing with the LZ4 example: if the user does not install the LZ4 library and filter, they cannot read LZ4 compressed data, and they get a filter error. Using ncdump -h -s they can find out the number of the missing filter. Then they can figure out how to install the library and filter and get the data. They will not need to recompile HDF5 or netcdf-c to do so, if they are using shared libraries. (@DennisHeimbigner please correct me if I am wrong about the filter API.)

What I would prefer is if HDF5 always included the LZ4 filter code, and then netcdf-c can require (as it does with zlib) or optionally support (as it does with szip) LZ4. In other words, the user must install zlib, and probably should install szip and lz4 libraries, before building HDF5 (and we can check this at netcdf-c configure time).

This would allow everyone to have LZ4 reliably on Linux, Max, and Windows (since all these packages support cmake builds on windows).

netcdf-java would then have to write or find an LZ4 decompressor (or switch to using HDF5 C library, which is probably inevitable anyway).

Any old versions of netCDF and HDF5 would have to upgrade to read the LZ4 compressed data (but that does not violate the backward compatibility guarantee).

So getting HDF5 to always ship the LZ4 filter code (and build it when liblz4 is found) would present a lot of advantages.

However, this is not an issue for szip. ;-)

[Dave-Allured]

@DennisHeimbigner wrote:

because there is no way for HDF5 to find and load the filter code
statically that I am aware of.

Agreed. My remark "in principle" was oversimplified, sorry. I meant that regardless of static or shared, if the user's set of installed libraries is missing something, they will still get a controlled runtime message along the lines of "unknown filter", either way. Then their remedy is also about the same, either way.

Ed's detailed reply of just now reinforces what I was trying to say, including the part about optional libraries in static builds.

[czender]

Expecting users to track down and install libraries for missing filters will doom filters to the limbo of the unreliable. Most filters of interest are freely redistributable and we should aim to provide a user-friendlier mechanism to guarantee their presence.

[Dave-Allured]

Expecting users to track down and install libraries for missing filters will doom filters to the limbo of the unreliable.

And yet, that is just what we expect these days for almost all other library dependencies in open source software, including the netcdf ecosystem. I just see filters as an extension of the same thing. I will now refrain from a litany of the pros and cons, trying to respect the current main topic.

My most important point is, defer filter management to HDF5. They have been doing it well for decades!

Most filters of interest are freely redistributable and we should aim to provide a user-friendlier mechanism

Several are already in place. Simple improvements could be made in some cases.

• List of sanctioned filters in build documentation (exists)
• Pointers to filter source repos in build documentation (missing)
• Improved install documentation
• Binary filter bundles (exists)
• Package managers (debian, anaconda, macports, etc.)

to guarantee their presence.

I substitute the resources above, plus user responsibility, for "guarantee". This becomes a matter of opinion about user behavior. We can just disagree for now.

[Dave-Allured]

... your points are valid for other filters, but none of this is necessary for the szip filter (the topic of this issue) - it already ships with HDF5, so is always present.

Ed, I searched recent HDF5 source distributions -- the C sources only. I was not able to find the szip library, only the HDF5 interface functions to that library; plus some precompiled binaries. It would seem that you really do need to separately install the szip library, at least for a full build from source. Am I missing something?

[DennisHeimbigner]

Dave is correct. The HDF code base does not contain any compressor
code, including zip and szip, It relies on externally supplied libraries.
All it provides is the wrapper code.

[dopplershift]

Expecting users to track down and install libraries for missing filters will doom filters to the limbo of the unreliable. Most filters of interest are freely redistributable and we should aim to provide a user-friendlier mechanism to guarantee their presence.

I agree, we need to make sure we provide an easy way for users to get their hands on the C and Java code to enable the filters they need.

[edwardhartnett]

To clarify, there are two bits of code involved in a compression filter like zlib, szip, and LZ4.

There is an external library (libz, libszip, liblz4). Then there is the HDF5 code that is needed to use the library and act as a filter (a.k.a the "filter code"). So the filter code is different from the external library. (And the filter code is small: the LZ4 filter code is only one file, ~250 lines of code, including comments.)

No one is suggesting that the netcdf or HDF5 teams take any responsibility for the external libraries. It will always be true that users must locate and install zlib before building HDF5. The same for szip, lz4, and any other compression method that uses an external library.

The filter code that uses zlib is already part of HDF5, so it builds with HDF5. The same for the szip filter code. If szip is found when HDF5 is built, then the filter for szip is built. But for other filters, like LZ4, the HDF5 filter code does not come with HDF5, it must be found, built, and installed separately by the user. Dealing with the HDF5 filter code is the extra step we want to eliminate.

What I am proposing is that the filter code for LZ4 be treated the same way as the szip filter - shipped and installed with HDF5 (when liblz4 is available at HDF5 build time). Then, at netcdf-c configure time, we can check whether HDF5 supports LZ4, and include support in netcdf-c, or else return the NC_ENOTBUILT error when the user tries to use LZ4.

No changes are needed in the HDF5 library to support this. We just need to change the HDF5 install systems a little, to install the LZ4 filter, when liblz4 is found on the machine.

This way, as with zlib and szip, the user will only have to be responsible installing the compression libraries and then building HDF5. There would be no additional step (as there is now) where the user has to get the LZ4 filter code, build it, and install it in a non-standard place to get it working.

I agree with @Dave-Allured that system admins and users are quite used to and capable of assembling the libraries needed. And package management systems like yum and apt can and will automate all this for most users.

But the filter code is something else - something non-standard. Let's try and solve that part for the user, for LZ4 (and perhaps other additional filters, if that seems desirable).

The filter code is really HDF5 code, and should ship and install with HDF5. But we should not be afraid of taking action ourselves, even if the HDF5 team does not ship the filter. These are small bits of code, and, where available, I certainly would not be terrified to take on support for them. They are mostly wrapper codes, and would require little or no additional work.

The idea of the Community Codec Repo (CCR) is that it will provide all the filter code, and the integration glue needed to get it working easily in netcdf-c. If the HDF5 team is reluctant to ship the LZ4 filter code, we can put it in CCR. The user will then have an extra step (to download and install CCR), but it will be a standard install, and will take care of all the HDF5 non-standard filter stuff automatically.

To the extent that we can package these solutions with HDF5 and netcdf-c, we will not have to put them in the CCR. Ideally, with LZ4 support in both HDF5 and netcdf-c, there would be no role for the CCR with LZ4.

[Dave-Allured]

Ed, thanks for clarifying. I misinterpreted the generic term "filter code", thus I confused the discussion of API's and filter interlude code vs. external libraries.

I would prefer a user-facing filter API that is generic and stable for all filters. The current nc_def_var_filter fits that, but there is room for improving ease of use. Unidata's objections to expanding the number of filter-specific API's is understandable. So for the time being, I support the idea of new specific filter API's as external code, as you are now doing with the CCR.

[Dave-Allured]

On the other hand, adding a handful of preferred filters with specific API's would be pretty reasonable, as you have previously said.