Skip to content

v3.4.0

Latest

Choose a tag to compare

@ayzk ayzk released this 04 Oct 20:59
592df79

New

  • One Huffman encoder for every algorithm, replacing the two earlier ones. Its code table takes about 0.3–0.7 bytes per symbol, where the one most algorithms used took 9–26. On 81 real fields from 32 datasets, the compression ratio is 1.5% higher and decompression 1.8 times faster (geometric means). #163
  • Every build decompresses to the same values, whether or not the compiler fuses multiply-adds (Apple Silicon, aarch64, -march=native, Spack's default) or uses fast math (Clang, icx and MSVC fast modes; GCC's -ffast-math and x87 are refused at compile time). #162, #166
  • An installed SZ3 works through find_package(SZ3). It provides SZ3::SZ3core (SZ3 and Zstd) and SZ3::SZ3 (plus OpenMP when SZ3 was built with it), and find_package(SZ3 COMPONENTS hdf5sz3) requires the HDF5 filter's SZ3::hdf5sz3. Added with add_subdirectory or FetchContent, SZ3 leaves the parent's build settings alone and installs nothing. #125, #126, #127, #128, #150, #160, #161
  • HDF5 filter: H5Pset_sz3() sets the filter from C. The filter's cd_values carry the data version, each chunk is decompressed with the configuration stored in it, and the filter does not use OpenMP. See tools/H5Z-SZ3/README.md. #155, #161
  • A build without OpenMP decompresses data compressed with OpenMP. #160
  • A Nix expression for NixOS. #129
  • ALGO_BIOMD is a new codec for molecular-dynamics coordinates {frames, atoms, 3} with an absolute bound: rigid water and bonded atoms are predicted from their geometry on an integer lattice. On 16 GROMACS trajectories at 5e-4 nm its compression ratio is 1.23 times that of the previous ALGO_BIOMD with one frame per chunk and 1.29 times with 20. Other shapes throw std::invalid_argument (in HDF5, creating the dataset fails); coordinates beyond its lattice (2048 nm at 5e-4 nm and 128 nm at 5e-5 nm for float) and NaN or Inf outside trailing fill frames go to lossless compression. #169
  • docs/molecular-dynamics.md for molecular-dynamics data, and the advanced guide for algorithms, building, CMake and packaging, the C++ API, OpenMP, floating point and contributing; the HDF5 filter's README covers using it in an application; the README is a quick start. #173
  • find_package(SZ3 COMPONENTS SZ3c) provides SZ3::SZ3c, the SZ2-compatible C library. A static one defines SZ3C_STATIC for its consumers, so it links on Windows. #177, #184
  • Homebrew: brew install szcompressor/tap/sz3. #176
  • pysz builds against an installed SZ3 and its Zstd with PYSZ_SZ3_PREFIX. #177
  • OpenMP splits ALGO_BIOMD data {frames, atoms, 3} along the atoms, so every part keeps all frames: with 16 threads and 20 frames per chunk, systems of 60,000 to 200,000 atoms lose at most 0.5% of the compression ratio, where splitting along the frames lost 3% to 14%. #182

Compatibility

  • Data format 3.4.0. Data from 3.3.2 decompresses, except data compressed with ALGO_NOPRED or the molecular-dynamics algorithms, which is refused. Data from 3.3.1 and earlier cannot be decompressed. #163
  • SZBioMDDecomposition and make_decomposition_biomd take no quantizer. #169
  • Data compressed by an earlier version built with FMA may exceed the error bound when decompressed by 3.4.0; decompress it with the build that compressed it. #162
  • CMake 3.19 or newer is required. find_package(SZ3 <version>) accepts only SZ3 versions with the same major version. #161
  • Removed: ArithmeticEncoder, RunlengthEncoder, SZTruncateCompressor and the preprocessors, which no build target used, the bundled ska hash map, and HuffmanEncoderV2. #150, #157, #163
  • Removed: ALGO_BIOMDXTC and its LGPL encoder, so SZ3 is BSD only; use ALGO_BIOMD. The value stays defined, and compressing with it throws std::invalid_argument (H5Pset_sz3 returns an error). Data compressed with it by 3.3.2 no longer decompresses. LinearQuantizer no longer takes a third argument. #183
  • Config::load and the lossless stages take the length of the buffer they read. #145
  • Integer data compressed by earlier versions decompresses only with those versions. 8-byte integers beyond ±2^53 are refused. #165
  • cmake --install no longer installs sz3_customized_demo or share/SZ3/testfloat_8_8_128.dat.
  • A system Zstd is found with find_library (SZ3_ZSTD_LIBRARY), as an installed SZ3Config.cmake finds it, not with pkg-config, so SZ3 added as a subproject leaves the parent's ZSTD_* cache entries alone. #184
  • libhdf5sz3 and libSZ3c export only their public functions, and carry an SOVERSION. #158, #159
  • A static libhdf5sz3 keeps H5Pset_sz3, set_SZ3_conf_to_H5 and get_SZ3_conf_from_H5 hidden in the shared library it is linked into, so they cannot take over another SZ3 copy's calls. H5PLget_plugin_type and H5PLget_plugin_info stay exported, as HDF5's H5PLextern.h declares them. #174
  • 32-bit builds (wasm32) and 64-bit builds decompress each other's data: every size in the stream is 8 bytes, as 64-bit builds already wrote it. #177
  • Config::load skips fields that a later version appends to the end of a config. #178

Fixes

  • Damaged or truncated compressed data makes decompression throw instead of reading past the buffer, and the HDF5 filter reports the error instead of ending the program. #121, #145, #149
  • OpenMP: compression no longer writes an undecodable stream or hangs when a region gets fewer threads than asked or a chunk fails. #149, #161
  • ALGO_LORENZO_REG could write past its buffer with a small BlockSize, and with only regression enabled it read past its buffer on 3D and 4D data. #151, #161
  • Integer data is compressed as floating point and rounded back, so values near the type's limits no longer come back past the bound, and REL and PSNR bounds work on data whose range the type cannot hold. #165
  • Float data could come back a fraction of an ulp past the bound: LinearQuantizer checked the error after rounding it to float. #166
  • The HDF5 filter refuses datatypes it cannot compress, instead of writing data that comes back wrong. #158
  • The HDF5 filter keeps H5Z_FLAG_OPTIONAL (as h5py sets it), and refuses a chunk whose size a filter before it changed. #163
  • SZ3 builds for wasm32 again. #117
  • An error bound that is negative or not finite (also a relative bound over data with Inf) compresses losslessly with every algorithm, as a bound of 0 did. #169
  • ALGO_BIOMD stores a chunk with NaN or Inf losslessly also when built with GCC's -funsafe-math-optimizations and -ffinite-math-only for an FMA target, which made it code NaN as 0. #171
  • cdvalueHelper.py writes cd_values with the data version first, as the filter reads them; mdz accepts 3D input in Debug builds and refuses 1D input, which it could not compress. #173
  • libSZ3c returns NULL on an error instead of ending the program with exit(0), and SZ_decompress checks the dimensions against the data. #177
  • Lossless_zstd::compress refuses a buffer under 8 bytes instead of writing past it. #177
  • ALGO_BIOMD refuses a corrupt mode byte or box flag instead of decoding with it. #178
  • Warnings under GCC 16. #178

Compared with other compressors

Single-threaded on an Intel i9-13900K (GCC 13, -O3). SZ3 uses its default ALGO_INTERP_LORENZO; every compressor runs in its pointwise error mode with the bound REL × (max − min). Geometric means over five float fields: Hurricane P, Miranda density, NYX temperature, SCALE QV and CESM U. Speeds are in original bytes per second, with 1 MB = 10^6 bytes. Versions: ZFP develop c0c2c40, SPERR 0.8.4 4061a44, MGARD 1.7.0 6b43196 (MGARD is mgard::compress, MGARD-X its serial device), SZx 3a0f875, SZp 5d15dc3. ZFP and MGARD stay at 2–70% of the bound, which lowers their ratios.

Compression ratio REL 1e-2 REL 1e-3 REL 1e-4
SZ3 3.4.0 1295.5 150.5 32.2
SPERR 1026.6 179.3 46.2
ZFP 26.6 13.0 6.9
MGARD 12.2 6.0 4.0
MGARD-X 20.3 12.0 6.6
SZx 21.4 6.5 4.0
SZp 36.4 14.0 8.5
Compression speed (MB/s) REL 1e-2 REL 1e-3 REL 1e-4
SZ3 3.4.0 507 474 434
SPERR 153 137 108
ZFP 493 377 300
MGARD 56 57 54
MGARD-X 93 93 92
SZx 1706 1179 1124
SZp 3246 1630 1261
Decompression speed (MB/s) REL 1e-2 REL 1e-3 REL 1e-4
SZ3 3.4.0 1495 1197 980
SPERR 299 268 214
ZFP 1667 1109 749
MGARD 49 44 38
MGARD-X 92 89 84
SZx 2873 1716 1584
SZp 7161 3583 2601

What's Changed

  • Introduce SZ3::unordered_map for wasm32 compatibility by @juntyr in #117
  • point sz3-rs to upstream repository by @rroohhh in #118
  • Fix unsafe pointer arithmetic/unsigned wrapping by @ayzk in #121
  • fix path issue in cmake by @ayzk in #124
  • feat: Improve HDF5-SZ3 CMake export by @ayzk in #125
  • fix: export hdf5sz3 target and resolve dependencies for find_package consumers by @ayzk in #126
  • cmake: expose hdf5 target as SZ3::hdf5sz3 in build and export by @ayzk in #128
  • Add basic nix expression for use on NixOS. by @juselius in #129
  • Drop 31 no-op ALWAYS_INLINE keywords + fix SDRBench dataset URLs by @ayzk in #140
  • fix: propagate HDF5_IS_PARALLEL to parent scope for SZ3Config by @ayzk in #127
  • Bounds-check decompression against corrupted input by @ayzk in #145
  • Fix two ALGO_BIOMD defects, and run the MD algorithms on MD data by @ayzk in #147
  • Stop a bad input ending the caller's process, and keep OpenMP inside its own call by @ayzk in #149
  • Make an installed SZ3 usable by its consumers, and fix what that exposed by @ayzk in #150
  • Make size_est() a bound its save() honours, and put a sanitizer in CI by @ayzk in #151
  • Let MinGW install HDF5 again, and notice if it does not by @ayzk in #152
  • Say what 3.3.3 changed, without narrating one bug by @ayzk in #153
  • update README by @ayzk in #154
  • Version the filter's cd_values, give it a C interface, and drop its registration functions by @ayzk in #155
  • Remove modules that no build target uses by @ayzk in #157
  • Export only the public functions from libhdf5sz3 and libSZ3c by @ayzk in #159
  • Refuse datatypes the HDF5 filter cannot compress; version the shared libraries by @ayzk in #158
  • Add SZ3::SZ3core, and read OpenMP streams without OpenMP by @ayzk in #160
  • Prepare 3.4.0: OpenMP, Zstd, HDF5 filter and packaging fixes by @ayzk in #161
  • Decode alike in builds that do and do not fuse multiply-adds by @ayzk in #162
  • Rewrite the README for first-time users; version history in CHANGELOG.md by @ayzk in #164
  • Compress integer data as floating point, and round it back by @ayzk in #165
  • One Huffman encoder for every algorithm; V1 kept for 3.3.2 streams by @ayzk in #163
  • 3.4.0 pre-release fixes: stream version on recompression, H5Z README, install, Zstd fallback by @ayzk in #166
  • pysz 1.1.0 on SZ3 3.4.0 by @ayzk in #167
  • 3.4.0: comparison with other compressors in the CHANGELOG; install sz3_smoke_test, not the sample data by @ayzk in #168
  • ALGO_BIOMD: geometry-aware codec for molecular-dynamics coordinates by @ayzk in #169
  • CMake: build SZ3's tests only when SZ3 is the top-level project by @ayzk in #170
  • BIOMD: test for NaN and Inf on the bits by @ayzk in #171
  • H5Z-SZ3: build the filter's tests only when SZ3 is the top-level project by @ayzk in #172
  • 3.4.0: a quick-start README, an advanced guide and an MD page; release line by @ayzk in #173
  • H5Z-SZ3: keep a static filter's functions hidden by @ayzk in #174

New Contributors

Full Changelog: v3.3.2...v3.4.0