Flint

Pure-Crystal LZ4 block codec.

flint implements LZ4 blocks with optional raw-content dictionaries. Public BlockCodec#compress prepends a 4-byte little-endian decoded-size header before the raw LZ4 block. #compress_raw and #decompress_raw expose raw LZ4 blocks for interoperability. The shard does not use FFI, liblz4, or a Rust extension.

Dictionary#id is metadata for callers that need to tag dictionary bytes in their own envelope. Raw blocks do not serialize or check the dictionary ID.

API

require "flint"

codec = Flint::BlockCodec.new
plain = "hello hello hello".to_slice

compressed = codec.compress(plain)
decoded = codec.decompress(compressed, max_size: plain.size)
decoded = codec.decompress(compressed)

Dictionary use:

dict = Flint::Dictionary.new("common prefix: ".to_slice)
codec = Flint::BlockCodec.new(dict)

msg = "common prefix: payload".to_slice
compressed = codec.compress(msg)
decoded = codec.decompress(compressed, max_size: msg.size)

Performance

flint is a pure-Crystal block codec. Expect portable speed, not native liblz4 speed.

Local reference run with Crystal 1.21.0, --release, on an Intel Core i7-8700B:

Workload	Compress	Decompress
Silesia geomean	463 MB/s	1,238 MB/s
Compressible Silesia files	190-533 MB/s	445-1,388 MB/s
Mostly incompressible Silesia files	1,041-4,436 MB/s	5,103-10,590 MB/s

Small repeated blocks run in hundreds of nanoseconds to low microseconds. In the self benchmark, a 1 MiB repeated-text block compressed in about 159 us and decompressed in about 98 us without a dictionary. Dictionary mode can improve small-payload ratios when the prefix matches, but it costs more CPU on larger payloads.

Benchmarks

Self benchmark:

crystal run --release bench/flint_block.cr

Silesia corpus benchmark:

sh bench/silesia_compare.sh

The Silesia files download lazily into ignored corpus/silesia/. The benchmark reports compression and decompression MB/s per file plus geomean.

Optional comparison against naqvis/lz4.cr lives in bench/compare_lz4_cr. It uses that shard's frame API, because that shard is a liblz4 frame binding rather than a raw block codec.

cd bench/compare_lz4_cr
shards install
crystal run --release src/compare.cr

To compare the 84codes/lz4.cr fork, change the lz4 dependency in bench/compare_lz4_cr/shard.yml from naqvis/lz4.cr to 84codes/lz4.cr.