Pcodec (or Pco) losslessly compresses and decompresses numerical sequences with high compression ratio and moderately fast speed.
Use cases include:
- columnar data
- long-term time series data
- tensor data
- serving numerical data to web clients or edge devices
Data types:
u8, u16, u32, u64, i8, i16, i32, i64, f16, f32, f64
- CLI (also runs benchmarks)
- Rust API
- Python API
- Java API
- C API
Pco is designed specifically for numerical data, whereas alternatives rely on general-purpose (LZ) compressors that target string or binary data. Pco uses a holistic, 3-step approach:
- modes.
Pco identifies an approximate structure of the numbers called a
mode and then uses it to split numbers into latents.
As an example, if most numbers in a chunk are approximately multiples of 777,
int mult mode splits each number
xinto latent variablesl_0andl_1such thatx = 777 * l_0 + l_1. Most natural data uses classic mode, which simply matchesx = l_0. - delta encoding. Pco identifies whether certain latent variables would be better compressed as deltas between consecutive elements (or deltas of deltas, or deltas with lookback). If so, it takes differences.
- binning. This is the core and most novel part of Pco. Pco represents each delta-encoded latent variable as an approximate, entropy-coded bin paired with an exact offset into that bin. This nears the Shannon entropy of any smooth distribution very efficiently.
These 3 steps cohesively capture most entropy of numerical data without waste. In contrast, LZ compressors are only very effective for patterns like repeating exact sequences of numbers.
Pco is designed to embed into wrapping formats. To that end, we provide a wrapped API that allows users to place each of Pco's components in their files as needed. This separates concerns: Pco handles compression, and the wrapping format handles domain-specific needs like nullability, multiple columns/channels, random access, or seeking.
The standalone format is a minimal implementation of a wrapped format. It supports batched decompression and nothing else. It is mainly recommended for quick proofs of concept and benchmarking.
Pco has a hierarchy of multiple batches per page; multiple pages per chunk; and multiple chunks per file. By default Pco uses up to 2^18 (~262k) numbers per chunk if available.
| unit of ___ | size for good compression | |
|---|---|---|
| chunk | compression | >10k numbers |
| page | interleaving w/ wrapping format | >1k numbers |
| batch | decompression | 256 numbers (fixed) |
Compression will be disappointing if the data in a single chunk
- combines semantically different sequences,
- contains too few numbers (see above), or
- is inherently 2D or higher.
Example: the NYC taxi dataset has f64 columns for fare and
trip_miles.
Suppose we assign these as fare[0...n] and trip_miles[0...n] respectively, where
n=50,000.
- separate chunk for each column => good compression
- single chunk
fare[0], ... fare[n-1], trip_miles[0], ..., trip_miles[n-1]=> bad compression - single chunk
fare[0], trip_miles[0], ..., fare[n-1], trip_miles[n-1]=> bad compression