Random access decompression, using zstd or lz4.

General

Requires zstd built with multithreading support for operations with >1 workers.

Build

Using meson:

meson setup --buildtype builddir
meson compile -C builddir
# Run unit tests
meson test -C builddir
# Build Doxygen documentation
meson compile -C builddir doc

or using autotools:

autoreconf --install  # requires autoconf-archive
./configure
make

Test

Elementary test to compress, decomporess and validate a user-specified file. See test/example.c.

./example --zstd|--lz4 <path-to-uncompressed-file>

Benchmark

Elementary compression-only benchmark on a user-specified file. Allows controlling various parameters and measures time and resource usage. Tries to eliminate I/O variability by loading the whole file to memory at startup. See test/benchmark.c.

For a single run:

./benchmark --zstd|--lz4 <path-to-uncompressed-file> <workers> <frame-size>

For multiple runs:

# See/edit benchmark.sh for the # of workers/min frame sizes to test
./benchmark.sh --zstd|--lz4 <path-to-uncompressed-file> | tee report.txt
./report.awk

TODO

• More tests: standalone, multi-threaded.
• Pluggable memory management.
• Dictionaries?
• OPT: Stop relying on zstd's multi-threading?
  • Pro: could use vanilla-built zstd.
  • Pro: total control over the threading.
  • Con: no intra-frame parallelism.