Benchmark: Quantile Compression

[powturbo]

I've integrated quantile compression into Turbobench Compression Benchmark
using the ffi c-interface for 32/64 bits integers.
Executables for windows and linux can be downloaded from releases.
I've made also a quick test with 32-bits lz77 offsets:

turbobench -ezstd,9/qcomp32,8 silesia_o.u32

      C Size  ratio%     C MB/s     D MB/s    Name            File
    30556963    55.6      91           260    qcomp32 8       silesia_o.u32
    33077995    60.1       7          1048    zstd 15         silesia_o.u32
    33242362    60.4      61          1040    zstd 9          silesia_o.u32

More benchmarks are now available.

[mwlon]

Summary: at the chosen compression levels, Quantile Compression compresses much faster and gets better compression ratio in most facets, but notably gets a worse ratio on decimal floats and interleaved or shifting distributions. Quantile compression also decompresses much more slowly. I have been aware of these weaknesses and expect to address the compression ratio issues and part of the decompression speed issue in a future format (though I do not plan to compete on the highest decompression speeds).

@powturbo Is there an explanation of each dataset used here? Also, I don't see version information - would appreciate if you could point me to that or add it.

Closing the issue as there is no action required.

[powturbo]

I hope you'll continue to improve your excellent quantile compression. I see a lot of potential in improving the decompression speed.

• lz77 offsets are the output of lz77 compression algos like zstd. This is in general a smooth increasing distribution. Large integers are more common.
• The other datasets are described in the paper
  Large-Alphabet Semi-Static Entropy Coding Via Asymmetric Numeral Systems
  
  I prefer to use real datasets used in academia in order to facilitate a comparison.