Sharing benchmark observations on decimal floating-point compression in columnar analytics: comparing integer mapping with Gorilla/Chimp

[x-at-01]

In real-time analytical databases and MPP columnar engines, floating-point numeric columns (such as business metrics, financial figures, sensor telemetry, and latency stats) consume significant storage volume and memory bandwidth during column scans. Traditional floating-point codecs such as Gorilla, Chimp, and Patas rely on bitwise XOR differences and sequential bitstreams. These variable bit-alignments inherently impose serial dependencies during decoding, hindering SIMD vectorization across vector lanes.

Over the past months, I have been working on fastalp (available on crates.io), an implementation of the Adaptive Lossless Floating-Point (ALP) compression algorithm. Instead of bitwise XOR operations, it samples block values to identify the optimal decimal exponent e, scales numbers into exact integers via multiplication by 10^e, and compresses them using Frame-of-Reference (FOR) bit-packing.

Key architectural characteristics:

• Lossless decimal integer mapping: Eliminates floating-point mantissa noise losslessly, routing rare unrepresentable values to an auxiliary exception vector.
• Pure register SIMD decoding: Because integers are stored at uniform bit widths per block, decoding unpacks full SIMD registers in parallel without bitstream branching, achieving 55 to 77 GB/s decompression throughput.
• Zero heap allocation: Exposes compress_into and decompress_into methods for in-place buffer execution within vectorized batch pipelines.
• Guaranteed bound: Reverts to raw uncompressed bytes when data entropy is high, preventing compressed size explosion.

Benchmark results measured on standard time-series datasets (weather telemetry, stock metrics, disk monitoring) with 1,000 double-precision values per batch:

Algorithm	Compressed Size (bits/val)	Compression Latency (µs / 1000 vals)	Decompression Latency (µs / 1000 vals)	Decompression Throughput
fastalp	16.34	2.255	0.423	2364 Mval/s
Chimp128	17.29	8.631	9.270	108 Mval/s
Patas	21.51	3.820	5.140	195 Mval/s
Gorilla	52.70	6.042	5.920	169 Mval/s

In these measurements, fastalp achieved 16.34 bits/val while completing decompression in 0.423 µs per 1,000 values (over 2.3 billion values per second on a single core).

I hope these benchmark figures and architectural observations offer useful context for columnar vector format evaluations and compression discussions in Doris.

[x-at-01]

Performance update: fastalp v0.1.37 release

I have updated the benchmark observations with the latest fastalp v0.1.37 release (published on crates.io with source code at fastalp).

Key improvements and updated metrics:

1. Decompression throughput: reached 32.53 GB/s (a 14.7% increase from 28.36 GB/s), compared to reference C++ ALP at ~20.0 GB/s and Gorilla / Chimp decoders at 1.5 ~ 2.5 GB/s. The speedup comes from zero-cost monomorphized decoder traits and direct raw pointer writes without temporary slice allocations.

2. Compression ratio: across the 37 standard time-series datasets from the SIGMOD 2024 ALP benchmark, the total compressed volume dropped from 104,465 bytes to 93,909 bytes (a 10.1% footprint reduction and +11.2% compression ratio improvement, reaching 3.23x overall and 6.99x geometric mean). Smooth datasets with continuous trends reach up to 431x via relaxed delta threshold evaluation.

3. Compression throughput: 4.87 GB/s with full dynamic parameter sampling, and 6.02 GB/s for pure kernel encoding with cached parameters (vs 5.45 GB/s for reference C++ ALP).

4. Production-grade memory soundness: full zero-allocation slice APIs (compress_into and decompress_into_raw) with bounded expansion on incompressible streams.