Adding Blosc2 HDF5 filter

README.rst

@@ -4,7 +4,7 @@ hdf5plugin
 .. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.7257761.svg
    :target: https://doi.org/10.5281/zenodo.7257761
 
-*hdf5plugin* provides `HDF5 compression filters <https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins>`_ (namely: Blosc, BitShuffle, BZip2, FciDecomp, LZ4, SZ, SZ3, Zfp, ZStd) and makes them usable from `h5py <https://www.h5py.org>`_.
+*hdf5plugin* provides `HDF5 compression filters <https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins>`_ (namely: Blosc, Blosc2, BitShuffle, BZip2, FciDecomp, LZ4, SZ, SZ3, Zfp, ZStd) and makes them usable from `h5py <https://www.h5py.org>`_.
 
 See `documentation <http://www.silx.org/doc/hdf5plugin/latest/>`_.
 

doc/contribute.rst

@@ -133,6 +133,34 @@ compression_opts: (0, 0, 0, 0, **compression level**, **shuffle**, **compression
 
 By default the filter uses byte shuffle and blosclz.
 
+blosc2
+......
+
+compression_opts: (0, 0, 0, 0, **compression level**, **filter**, **compression**)
+
+- First 4 values are reserved.
+- **compression level**:
+  From 0 (no compression) to 9 (maximum compression).
+  Default: 5.
+- **filter**: Pre-compression filter:
+
+  * 0: no shuffle
+  * 1: byte shuffle
+  * 2: bit shuffle
+  * 3: delta: diff current block with first one
+  * 4: truncate precision: Truncate mantissa for floating point types
+
+- **compression**: The compressor blosc ID:
+
+  * 0: blosclz (default)
+  * 1: lz4
+  * 2: lz4hc
+  * 3: unused
+  * 4: zlib
+  * 5: zstd
+
+By default the filter uses byte shuffle and blosclz.
+
 bzip2
 .....
 

doc/index.rst

@@ -1,7 +1,8 @@
 hdf5plugin
 ==========
 
-*hdf5plugin* provides `HDF5 compression filters <https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins>`_ (namely: Blosc, BitShuffle, BZip2, FciDecomp, LZ4, SZ, SZ3, Zfp, ZStd) and makes them usable from `h5py <https://www.h5py.org>`_.
+*hdf5plugin* provides `HDF5 compression filters <https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins>`_ (namely: Blosc, Blosc2, BitShuffle, BZip2, FciDecomp, LZ4, SZ, SZ3, Zfp, ZStd) and makes them usable from `h5py <https://www.h5py.org>`_.
+
 
 * Supported operating systems: Linux, Windows, macOS.
 * Supported versions of Python: >= 3.7

doc/information.rst

@@ -53,6 +53,7 @@ HDF5 compression filters and compression libraries sources were obtained from:
 * **bitshuffle plugin** (0.4.2 + patch `PR #122 <https://github.com/kiyo-masui/bitshuffle/pull/122>`_) and **zstd** (v1.5.2): https://github.com/kiyo-masui/bitshuffle and https://github.com/Blosc/c-blosc/tree/9dc93b1de7c1ff6265d0ae554bd79077840849d8/internal-complibs/zstd-1.5.2
 * **bzip2 plugin** (from PyTables v3.7.0) and **bzip2** (v1.0.8): https://github.com/PyTables/PyTables/, https://sourceware.org/git/bzip2.git
 * **hdf5-blosc plugin** (v1.0.0), **c-blosc** (commit `9dc93b1 <https://github.com/Blosc/c-blosc/tree/9dc93b1de7c1ff6265d0ae554bd79077840849d8>`_) and **snappy** (v1.1.9): https://github.com/Blosc/hdf5-blosc, https://github.com/Blosc/c-blosc and https://github.com/google/snappy
+* **hdf5-blosc2 plugin** (from PyTables commit `8b8c7bc <https://github.com/PyTables/PyTables/commit/8b8c7bc7b1ff7f0a17bdd8b9f07198ab1bb4666d>`_), **c-blosc2** (`v2.6.0 <https://github.com/Blosc/c-blosc2/releases/tag/v2.6.0>`_ + `patch <https://github.com/Blosc/c-blosc2/pull/431>`_): https://github.com/PyTables/PyTables/, https://github.com/Blosc/c-blosc2
 * **FCIDECOMP plugin** (v1.0.2) and **CharLS** (1.x branch, commit `25160a4 <https://github.com/team-charls/charls/tree/25160a42fb62e71e4b0ce081f5cb3f8bb73938b5>`_):
   ftp://ftp.eumetsat.int/pub/OPS/out/test-data/Test-data-for-External-Users/MTG_FCI_Test-Data/FCI_Decompression_Software_V1.0.2 and
   https://github.com/team-charls/charls
@@ -73,6 +74,7 @@ Please read the different licenses:
 
 * bitshuffle: See `src/bitshuffle/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/bitshuffle/LICENSE>`_
 * blosc: See `src/hdf5-blosc/LICENSES/ <https://github.com/silx-kit/hdf5plugin/blob/main/src/hdf5-blosc/LICENSES/>`_, `src/c-blosc/LICENSES/ <https://github.com/silx-kit/hdf5plugin/blob/main/src/c-blosc/LICENSES/>`_ and `src/snappy/COPYING <https://github.com/silx-kit/hdf5plugin/blob/main/src/snappy/COPYING>`_
+* blosc2: See `src/PyTables/LICENSE.txt <https://github.com/silx-kit/hdf5plugin/blob/main/src/PyTables/LICENSE.txt>`_  and `src/c-blosc2/LICENSES/ <https://github.com/silx-kit/hdf5plugin/blob/main/src/c-blosc2/LICENSES/>`_
 * bzip2: See `src/PyTables/LICENSE.txt <https://github.com/silx-kit/hdf5plugin/blob/main/src/PyTables/LICENSE.txt>`_ and `src/bzip2/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/bzip2/LICENSE>`_
 * lz4: See `src/LZ4/COPYING <https://github.com/silx-kit/hdf5plugin/blob/main/src/LZ4/COPYING>`_, `src/LZ4/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/LZ4/LICENSE>`_ and `src/c-blosc/LICENSES/LZ4.txt <https://github.com/silx-kit/hdf5plugin/blob/main/src/c-blosc/LICENSES/LZ4.txt>`_
 * FCIDECOMP: See `src/fcidecomp/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/fcidecomp/LICENSE.txt>`_ and `src/charls/src/License.txt  <https://github.com/silx-kit/hdf5plugin/blob/main/src/charls/src/License.txt>`_

doc/usage.rst

@@ -62,6 +62,13 @@ Blosc
    :members:
    :undoc-members:
 
+Blosc2
+======
+
+.. autoclass:: Blosc2
+   :members:
+   :undoc-members:
+
 BZip2
 =====
 

setup.py

@@ -775,6 +775,72 @@ def get_blosc_plugin():
 PLUGIN_LIB_DEPENDENCIES['blosc'] = 'snappy', 'lz4', 'zlib', 'zstd'
 
 
+def get_blosc2_plugin():
+    """blosc2 plugin build config
+
+    Source from PyTables and c-blosc2
+    """
+    hdf5_blosc2_dir = 'src/PyTables/hdf5-blosc2/src'
+    blosc2_dir = 'src/c-blosc2'
+
+    # blosc sources
+    sources = [f for f in glob(f'{blosc2_dir}/blosc/*.c')
+            if 'avx2' not in f and 'sse2' not in f and 'altivec' not in f and 'neon' not in f]
+    include_dirs = [blosc2_dir, f'{blosc2_dir}/blosc', f'{blosc2_dir}/include']
+    define_macros = []
+
+    # TODO enable neon
+    sse2_kwargs = {
+        'sources': glob(f'{blosc2_dir}/blosc/*-sse2.c'),
+        'define_macros': [('SHUFFLE_SSE2_ENABLED', 1)],
+        }
+
+    avx2_kwargs = {
+        'sources': glob(f'{blosc2_dir}/blosc/*-avx2.c'),
+        'define_macros': [('SHUFFLE_AVX2_ENABLED', 1)],
+        }
+
+    if platform.machine() == "ppc64le":  # altivec
+        sources += glob(f'{blosc2_dir}/blosc/*-altivec.c')
+        define_macros += [('SHUFFLE_ALTIVEC_ENABLED', 1)]
+
+
+    # compression libs
+    # lz4
+    include_dirs += get_lz4_clib('include_dirs')
+    define_macros.append(('HAVE_LZ4', 1))
+
+    # zlib
+    include_dirs += get_zlib_clib('include_dirs')
+    define_macros.append(('HAVE_ZLIB', 1))
+
+    # zstd
+    include_dirs += get_zstd_clib('include_dirs')
+    define_macros.append(('HAVE_ZSTD', 1))
+
+    extra_compile_args = ['-std=gnu99']  # Needed to build manylinux1 wheels
+    extra_compile_args += ['-O3', '-ffast-math']
+    extra_compile_args += ['/Ox', '/fp:fast']
+    extra_compile_args += ['-pthread']
+    extra_link_args = ['-pthread']
+
+    return HDF5PluginExtension(
+        "hdf5plugin.plugins.libh5blosc2",
+        sources=sources + \
+            prefix(hdf5_blosc2_dir, ['blosc2_filter.c', 'blosc2_plugin.c']),
+        extra_objects=get_zstd_clib('extra_objects'),
+        include_dirs=include_dirs + [hdf5_blosc2_dir],
+        define_macros=define_macros,
+        extra_compile_args=extra_compile_args,
+        extra_link_args=extra_link_args,
+        sse2=sse2_kwargs,
+        avx2=avx2_kwargs,
+        )
+
+
+PLUGIN_LIB_DEPENDENCIES['blosc2'] = 'lz4', 'zlib', 'zstd'
+
+
 def get_zstandard_plugin():
     """HDF5Plugin-Zstandard plugin build config"""
     zstandard_dir = 'src/HDF5Plugin-Zstandard'
@@ -1031,6 +1097,7 @@ def apply_filter_strip(libraries, extensions, dependencies):
         get_lz4_plugin(),
         get_bitshuffle_plugin(),
         get_blosc_plugin(),
+        get_blosc2_plugin(),
         get_fcidecomp_plugin(),
         get_h5zfp_plugin(),
         get_zstandard_plugin(),