Adding SZ?

[orioltinto]

Hi,
I'm currently working on integrating lossy+lossless compression in our python tools through hdf5 filters.

I'm really happy with this hdf5plugin library and how easy is to get everything running.

However, in my research I've found that a very interesting compressor that also has a hdf5 filter available is SZ. In my research I found that in many cases it is the best performing compressor. Hopefully a publication would be out soon.

I would be happy to contribute adding this compressor to the list of available filters.

Would that be interesting for you?

Could someone give some feedback about how to use the same build system that already works for the other plugins? I gave a try but I was having problems.

By the way, I would be happy to cite this library in my publication, is there any proper citation?

Thanks in advance.

[t20100]

Hi,

Sure! The goal of this project is to ease (and so promote) usage of HDF5 compression filters with h5py, so SZ is welcomed as well as contributions!

To fit into hdf5plugin, there are a few constraints:

• First is open source license (should be OK here),
• Then is the possibility to build the plugin in a "portable" way in the sense of allowing to provide pre-built binaries (Python wheels, conda packages) for Windows, Linux, macOS. The strategy so far was to allow such pre-built binary but also to enable as much optimisation as possible (SSE2, AVX2, OpenMP...) when building from source.
• Finally constraint is to be able build the plugin from the setup.py, which can be a bit complicated because it needs to understand how the filter build works.

There is some general guidelines explaining how to add a compression filter here.
For example for the lz4 filter, it is defined here:

hdf5plugin/setup.py

Lines 681 to 695 in 80e2f7d

	# lz4 plugin
	# Source from https://github.com/nexusformat/HDF5-External-Filter-Plugins
	if sys.platform.startswith('darwin'):
	extra_compile_args = ['-Wno-error=implicit-function-declaration']
	else:
	extra_compile_args = []
	lz4_plugin = HDF5PluginExtension(
	"hdf5plugin.plugins.libh5lz4",
	sources=['src/LZ4/H5Zlz4.c', 'src/LZ4/lz4_h5plugin.c'] + lz4_sources,
	depends=lz4_depends,
	include_dirs=lz4_include_dirs,
	extra_compile_args=extra_compile_args,
	libraries=['Ws2_32'] if sys.platform.startswith('win') else [],
	)

and added to the list of filters to build here:

hdf5plugin/setup.py

Lines 831 to 839 in 80e2f7d

	extensions=[
	bzip2_plugin,
	lz4_plugin,
	bithsuffle_plugin,
	blosc_plugin,
	fcidecomp_plugin,
	h5zfp_plugin,
	zstandard_plugin,
	],

The filters are built using a patched Python extension build from setuptools:

hdf5plugin/setup.py

Line 476 in 80e2f7d

class HDF5PluginExtension(Extension):

If you start working on this, we can provide support along the way!

Regarding citation, there is no publication about this project. However, I just added hdf5plugin to Zenodo to have DOIs, so there is now a "project" DOI to reference all versions and a DOI for the latest version 3.3.1. Upcoming versions will also get a DOI.

Looking forward for your publication! I'm interested in compressors' comparison.

[orioltinto]

Thanks @t20100.

First is open source license (should be OK here),

SZ licence is under the BSD license.

Then is the possibility to build the plugin in a "portable" way in the sense of allowing to provide pre-built binaries (Python wheels, conda packages) for Windows, Linux, macOS. The strategy so far was to allow such pre-built binary but also to enable as much optimisation as possible (SSE2, AVX2, OpenMP...) when building from source.

I believe it should be possible.

Finally constraint is to be able build the plugin from the setup.py, which can be a bit complicated because it needs to understand how the filter build works.

When I tried to imitate what was done for other filters I got a bit stuck.
How I usually compile SZ to include also the HDF5 filter is using CMake:

CMAKE_FLAGS="-DCMAKE_INSTALL_PREFIX:PATH=${INSTALLDIR} -DBUILD_HDF5_FILTER:BOOL=ON"
cmake .. ${CMAKE_FLAGS}

It believe it is also possible compile the tool without cmake but it is necessary to first execute the configure script.

Trying to mimic the way the other filters are installed, I end up with an error because the compiler can't find "config.h", which is created in configuration phase.

      src/SZ/sz/src/sz_double.c:10:10: fatal error: config.h: No such file or directory
         10 | #include "config.h"
            |          ^~~~~~~~~~

Any clue on whether it is feasible to use cmake or how to continue?

I've updated my changes in a fork:

https://github.com/orioltinto/hdf5plugin

Whenever I had a problem with SZ the developers were very responsive, I might ask them doubts as well.

[orioltinto]

Hi again,
I managed to build the SZ filter within hdf5plugins but I'm still having several problems:

• I built the filter using cmake, and doing it like this I can use it if I use HDF5_PLUGIN_PATH but it fails to load when the filters are initialized with hdf5plugin because a missing symbol init_filter.
• I tried to build it without cmake like the other existing filters but I had the problem that the sources depend on a file that is created when the configure script is called. Do you know any way of going around this? For the moment, if I bypass this by manually running configure before installing the package, I obtain the filter with init_filter symbol but when it is intialized complains about another missing symbol: undefined symbol: H5Tget_sign. Is it a problem with the version of HDF5?

[t20100]

Hi, great!

• For the config.h issue: I guess the ideal would be to use cmake for the build under the hood, but it is really far from how hdf5plugin has been working so far and put more constraint on the build from source. One way would be to provide an already built "generic" config.h file. Not ideal but simpler to integrate. The longer road would be to use cmake (as other embedded projects are using it too), but that needs a bit more evaluation to make sure to keep it easy to build from source on all platforms.
• For the init_filter issue, we add some code to each filter which provides this function: src/hdf5_dl.c. HDF5 filters are supposed to be linked with libhdf5 (i.e., they are not really plugins). It is not always possible to know in advance the libhdf5 used (e.g., when using h5py wheels which bundles their libhdf5). To workaround this issue, we instead link with stubs of HDF5 functions. Those functions are then calling the proper HDF5 functions that are dynamically loaded (using dlopen/dlsym) at runtime. This dynamic loading is done by this init_filter function.
• H5Tget_sign issue: It is a missing stub in src/hdf5_dl.c. We add them when needed. I can add it.

[t20100]

Here is a patch to add H5Tget_sign:

Patch...

+++ b/src/hdf5_dl.c
@@ -86,6 +86,7 @@ typedef htri_t (* DL_func_H5Sis_simple)(hid_t space_id);
 typedef herr_t (* DL_func_H5Tconvert)(hid_t src_id, hid_t dst_id, size_t nelmts,
     void *buf, void *background, hid_t plist_id);
 typedef hid_t (* DL_func_H5Tget_native_type)(hid_t type_id, H5T_direction_t direction);
+typedef H5T_sign_t (* DL_func_H5Tget_sign)(hid_t type_id);
 typedef size_t (* DL_func_H5Tget_size)(
     hid_t type_id);
 typedef H5T_class_t (* DL_func_H5Tget_class)(hid_t type_id);
@@ -123,6 +124,7 @@ static struct {
     /*H5T*/
     DL_func_H5Tconvert H5Tconvert;
     DL_func_H5Tget_native_type H5Tget_native_type;
+    DL_func_H5Tget_sign H5Tget_sign;
     DL_func_H5Tget_size H5Tget_size;
     DL_func_H5Tget_class H5Tget_class;
     DL_func_H5Tget_order H5Tget_order;
@@ -133,7 +135,7 @@ static struct {
 } DL_H5Functions = {
     NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
     NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
-    NULL, NULL, NULL, NULL, NULL, NULL};
+    NULL, NULL, NULL, NULL, NULL, NULL, NULL};
 
 
 /*HDF5 variables*/
@@ -211,6 +213,7 @@ int init_filter(const char* libname)
         /*H5T*/
         DL_H5Functions.H5Tconvert = (DL_func_H5Tconvert)dlsym(handle, "H5Tconvert");
         DL_H5Functions.H5Tget_native_type = (DL_func_H5Tget_native_type)dlsym(handle, "H5Tget_native_type");
+        DL_H5Functions.H5Tget_sign = (DL_func_H5Tget_sign)dlsym(handle, "H5Tget_sign");
         DL_H5Functions.H5Tget_size = (DL_func_H5Tget_size)dlsym(handle, "H5Tget_size");
         DL_H5Functions.H5Tget_class = (DL_func_H5Tget_class)dlsym(handle, "H5Tget_class");
         DL_H5Functions.H5Tget_order = (DL_func_H5Tget_order)dlsym(handle, "H5Tget_order");
@@ -418,6 +421,11 @@ hid_t H5Tget_native_type(hid_t type_id, H5T_direction_t direction)
 CALL(0, H5Tget_native_type, type_id, direction)
 }
 
+H5T_sign_t H5Tget_sign(hid_t type_id)
+{
+CALL(-1, H5Tget_sign, type_id)
+}
+
 size_t H5Tget_size(hid_t type_id)
 {
 CALL(0, H5Tget_size, type_id)

[t20100]

Trying your fork, I get:

Traceback (most recent call last):
  File "hdf5plugin/__init__.py", line 479, in _init_filters
    lib = _ctypes.CDLL(filename)
  File "/opt/local/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: hdf5plugin/plugins/libh5sz.so: undefined symbol: H5PLget_plugin_info

But H5PLget_plugin_info is declared in the filter...
Did you encounter this? I'll look at it again.

[orioltinto]

Hi @t20100 ,
thanks for the feedback.

In the current version of the fork the filter is compiled with cmake, however not only the actual filter is compiled with a different name (libhdf5sz.so) another "filter" is created. I guess it is because of the "bypass" I used to actually compile with cmake.
I believe that you can find that libhdf5sz.so and libSZ.so have been created in the plugins folder. When I tried I could use that filter if specifying HDF5_PLUGIN_PATH.

In my local branch I'm trying to build this filter like the other ones, I tried to use your patch and it solved the missing H5Tget_sign.
However, it failed with a different one and after checking with nm I see that I have a bunch other missing symbols. Some of those are coming from SZ. The filters use stuff from libSZ.so, do you know how that is handled in the other filters?

                 U H5Zunregister
                 U SZ_Init
                 U SZ_Init_Params
                 U SZ_compress
                 U SZ_compress_args
                 U SZ_decompress
                 U __printf_chk@GLIBC_2.3.4
                 U __stack_chk_fail@GLIBC_2.4
                 U __vsnprintf_chk@GLIBC_2.3.4
                 U bytesToDouble
                 U bytesToInt32_bigEndian
                 U bytesToInt_bigEndian
                 U bytesToLong_bigEndian
                 U calloc@GLIBC_2.2.5
                 U computeDataLength
                 U computeDimension
                 U confparams_cpr
                 U dataEndianType
                 U dlopen@GLIBC_2.34
                 U dlsym@GLIBC_2.34
                 U doubleToBytes
                 U exit@GLIBC_2.2.5
                 U filterDimension
                 U free@GLIBC_2.2.5
                 U int32ToBytes_bigEndian
                 U intToBytes_bigEndian
                 U longToBytes_bigEndian
                 U malloc@GLIBC_2.2.5
                 U memcpy@GLIBC_2.14
                 U stderr@GLIBC_2.2.5

[t20100]

For the other filters, compression libraries are statically compiled and linked with the filter to make a single .so.
From what is missing, only H5Zunregister is from HDF5.

I'm trying to build it with your previous approach, copying the config.h file in the source for now.

[orioltinto]

By my previous approach you mean cmake?

I updated my current changes in a branch, maybe it will be easier.

[t20100]

No I mean without cmake, with commit 356e7b5

Here is my current patch from commit 356e7b5

Patch...

--- a/setup.py
+++ b/setup.py
@@ -802,10 +802,17 @@ zfp_lib = ('zfp', {
 sz_dir = os.path.join("src", "SZ", "sz")
 sz_sources = glob(os.path.join(sz_dir, "src", "*.c"))
 sz_include_dirs = [os.path.join(sz_dir, "include"), sz_dir]
+sz_sources += glob('src/SZ/zstd/*/*.c')
+sz_sources += glob('src/SZ/zlib/*.c')
+# TODO sz_depends += glob('src/SZ/zstd*/*/*.h')
+sz_include_dirs += glob('src/SZ/zstd')
+sz_include_dirs += glob('src/SZ/zstd/common')
+sz_include_dirs += glob('src/SZ/zlib')
+
 sz_lib = ("sz", {
     "sources": sz_sources,
     "include_dirs": sz_include_dirs,
-    "cflags": ["-lzstd"],
+    #"cflags": ["-lzstd"],
 })
 
 # "cflags": ['-DBUILD_HDF5_FILTER:BOOL=ON'],
@@ -813,11 +820,11 @@ sz_lib = ("sz", {
 
 h5zsz_dir = os.path.join("src", "SZ", "hdf5-filter", "H5Z-SZ")
 sources = glob(h5zsz_dir + "/src/" + "*.c")
-depends = glob(h5zsz_dir + "/src/" + "*.h")
-include_dirs = [h5zsz_dir, sz_dir]
+depends = glob(h5zsz_dir + "/include/" + "*.h")
+include_dirs = [os.path.join(sz_dir, 'include'), os.path.join(h5zsz_dir, 'include')]
 extra_compile_args = ['-O3', '-ffast-math', '-std=c99', '-fopenmp']
 extra_compile_args += ['/Ox', '/fp:fast', '/openmp']
-extra_link_args = ['-fopenmp', '/openmp', "-lzstd"]
+extra_link_args = ['-fopenmp', '/openmp']
 
 sz_plugin = HDF5PluginExtension(
     "hdf5plugin.plugins.libh5sz",

And I get undefined symbol: inflate, from some reason zlib is not included.

[orioltinto]

I got it running*! (by manually running the config file before pip install ...)
The pending issue is how to deal with this config.h. Is providigin a generic file a proper solution or maybe I can try to somehow run the configure script from the setup.py?

[orioltinto]

Dockerfile to reproduce it:

FROM ubuntu:latest

# Install some dependencies through apt
RUN export DEBIAN_FRONTEND=noninteractive \
    && apt update \
    && apt install -yq git python3 python3-pip python3-venv

# Install hdf5plugin
RUN git clone https://github.com/orioltinto/hdf5plugin.git \
 && cd hdf5plugin/src/SZ ; ./configure; cd -               \
 && pip install /hdf5plugin

[t20100]

Great! I'll look at it.

[vasole]

Just a question. Can this filter be built under windows when using other compiler than mingw?