Sharing benchmark observations on decimal floating-point compression: comparing integer mapping with Gorilla/Chimp on time-series datasets

[x-at-01]

Hello DataFusion community,

Apache DataFusion is widely used across modern analytical and time-series query engines. Since floating-point columns represent a major workload in columnar analytics, I wanted to share benchmark observations and implementation findings on lossless floating-point compression from fastalp (available on crates.io), an implementation of the Adaptive Lossless Floating-Point Compression (ALP) algorithm.

Benchmark results on real-world time-series datasets

I evaluated fastalp against classic floating-point codecs (Gorilla, Chimp, Chimp128, Patas) on public time-series datasets including city_temperature.csv, Stocks-Germany-sample.txt, and SSD_HDD_benchmarks.csv. Running 25 iterations per dataset on Apple Silicon (M2 Max) produced the following unified averages:

Codec	Compression Ratio	Encoding Latency (µs / 1000 vals)	Decoding Latency (µs / 1000 vals)
fastalp	16.34 bits/val	2.255 µs	0.423 µs
Gorilla	52.70 bits/val	6.042 µs	5.920 µs
Patas	21.51 bits/val	6.818 µs	6.502 µs
Chimp128	17.29 bits/val	7.468 µs	7.637 µs
Chimp	41.08 bits/val	8.631 µs	9.270 µs

Overall throughput across standard IEEE 754 float benchmarks:

Implementation observations for columnar analytics

1. Decimal integer mapping: In real-world telemetry and business metrics, floating-point numbers often have a fixed or small number of decimal places (e.g. 19.99, 0.05). Mapping values to integers via adaptive decimal scaling (10^e) shifts the data domain to Frame-of-Reference (FOR) integer packing, enabling SIMD-accelerated execution rather than serial bitwise XOR streams.

2. Exact decimal division: Precomputed floating-point multiplication (value * 10^e) occasionally introduces 1-ULP precision noise due to IEEE 754 rounding (for example, 0.35 * 100.0 evaluates to 34.99999999999999, casting to 34 instead of 35). Adding an exact division path during sampling and reconstruction eliminates these pseudo-exceptions and preserves bit-packing efficiency.

3. First-order difference (Delta-ALP): For monotonic timestamps, cumulative metrics, and smooth waveforms, cascading first-order delta encoding onto the scaled integers significantly narrows residual values, improving compression ratios while preserving O(1) random block access.

4. Pure register SIMD decoding: Vectorized integer unpacking achieves 0.423 µs per 1000 values (~2.36 billion values/sec), reaching 55 to 77 GB/s linear memory bandwidth on modern CPUs.

5. Zero-heap streaming API: The compress_into and decompress_into interfaces allow encoding and decoding directly into caller-provided buffers, eliminating allocation overhead during query scans.

I hope these benchmark results and implementation findings provide useful reference points for columnar storage and query execution discussions.

[x-at-01]

Performance update: fastalp v0.1.37 release

I have updated the benchmark observations with the latest fastalp v0.1.37 release (published on crates.io with source code at fastalp).

Key improvements and updated metrics:

1. Decompression throughput: reached 32.53 GB/s (a 14.7% increase from 28.36 GB/s), compared to reference C++ ALP at ~20.0 GB/s and Gorilla / Chimp decoders at 1.5 ~ 2.5 GB/s. The speedup comes from zero-cost monomorphized decoder traits and direct raw pointer writes without temporary slice allocations.

2. Compression ratio: across the 37 standard time-series datasets from the SIGMOD 2024 ALP benchmark, the total compressed volume dropped from 104,465 bytes to 93,909 bytes (a 10.1% footprint reduction and +11.2% compression ratio improvement, reaching 3.23x overall and 6.99x geometric mean). Smooth datasets with continuous trends reach up to 431x via relaxed delta threshold evaluation.

3. Compression throughput: 4.87 GB/s with full dynamic parameter sampling, and 6.02 GB/s for pure kernel encoding with cached parameters (vs 5.45 GB/s for reference C++ ALP).

4. Production-grade memory soundness: full zero-allocation slice APIs (compress_into and decompress_into_raw) with bounded expansion on incompressible streams.