A series of experiments in compressing semi-structured floating point data. Blog post series about it:
- Part 0: Intro
- Part 1: Generic Compression Libraries (zlib, lz4, zstd, brotli)
- Part 2: Generic Compression Libraries (libdeflate, oodle)
- Part 3: Data Filtering (simple data filtering to improve compression ratio)
- Part 4: Mesh Optimizer (mis-using mesh compression library on our data set)
- Part 5: Science! (zfp, fpzip, SPDP, ndzip, streamvbyte)
- Part 6: Optimize Filtering (optimizations for part 3 data filters)
- Part 7: More Filtering Optimization (more optimizations for data filters)
Code here uses 3rd party libraries:
- zstd, v1.5.2, BSD or GPLv2 license.
- lz4, v1.9.3, BSD license.
- zlib, 1.2.13, zlib license.
- brotli, v1.0.9, MIT license.
- libdeflate, v1.17, MIT license.
- bitshuffle, 0.5.1, MIT license.
- meshoptimizer, v0.18, MIT license.
- spdp_11.c, BSD license.
- zfp, 1.0.0, BSD license.
- fpzip, 1.3.0, BSD license.
- ndzip, 2022.06, MIT license.
- streamvbyte, 0.5.2, Apache-2.0 license.
- sokol_time, zlib license.
- ini.h, 2022 Sep (aef7d92), MIT or Public Domain.
- lzsse, 2019 Apr 18 (1847c3e), BSD license.
- lizard, 2020 Nov 3 (af8518c), BSD license.
- sse2neon, 2023 Apr 14 (32ad00d), MIT license.
Oodle SDK is not public, so that one is not compiled in unless you put:
- Oodle wrapper functions implementation file into
src/oodle_wrapper.cpp
, with implementation ofoodle_init
,oodle_compress_calc_bound
,oodle_compress_data
,oodle_decompress_data
,oodle_compressor_get_version
. - Oodle header files and core static library under
libs/oodle
folder.