Support for DType::DecFloat

[moshap-firebolt]

Vortex supports Decimal - which is fixed precision numbers - same (precision, scale) for entire column
and it supports binary floating numbers.

We propose to add support for Decimal Floating (aka DecFloat) - decimals whose scale varies per value.

Such type exists in several data systems, but no columnar format atm can store them losslessly:

System	Type	Shape
MongoDB / BSON	Decimal128	IEEE 754-2008 decimal128, 34 digits, exponent ±6144
IBM Db2, Firebird	DECFLOAT(16|34)	IEEE decimal64 / decimal128 (SQL:2016)
Snowflake	DECFLOAT	38 digits, exponent −16383…16384
Trino 480+	NUMBER	200 digits, scale ±16384

Parquet's DECIMAL fixes scale per column; Arrow's Decimal32/64/128/256 likewise; ORC and Iceberg
the same. So these columns get rounded into a fixed (p,s), silently degraded to DOUBLE, or
stringified.

How the ecosystem copes today: MongoDB's own Spark connector infers the type per sampled value and
falls back to DataTypes.DoubleType when the merge would exceed precision 38 — silently turning a
monetary type into binary floating point. Trino's MongoDB connector samples to infer (p,s).
ClickHouse's MongoDB engine has no mapping at all.

Relevant prior art for the type itself:

• SQL:2016 DECFLOAT, defined as IEEE 754-2008 decimal64/decimal128 (COMPARE_DECFLOAT, TOTALORDER, NORMALIZE_DECFLOAT functions)
• BSON Decimal128 specification
• Db2 DECFLOAT,
  Firebird DECFLOAT

Logical DType

DType::DecFloat { max_digits, nullability }, where max_digits in {16, 34} corresponding to
IEEE decimal64 and decimal128. Both are fixed-width, so arrays stay fixed-width.

Canonical Array

A DecFloatArray holding a common exponent plus a coefficient array plus Patches for
values whose exponent differs from the common one:

DecFloatArray {
    exponent:    i16,          // common to the array
    coefficients: ArrayRef,    // signed integers, DecimalByteParts-compatible
    patches:     Option<Patches>,  // values with a different exponent
}

This is ALP's structure applied to decimals - the encoder signature Vortex already has:

alp_encode(...) -> VortexResult<(Exponents, ArrayRef, Option<Patches>)>

ALP exists because most real f64 values are decimal-shaped, so it factors out a common decimal
exponent and stores integers. For a decimal-float type that isn't a heuristic — it's the type's
literal semantics.

Two ways the decimal case is simpler than ALP for floats:

1. ALP must search for an exponent that round-trips and verify per value. Here the exponent is a
   field of the value, so "does this match the common exponent?" is a comparison, not a trial
   encoding. Compression is one pass, no search.
2. The decomposition is exact by construction — there's no "does this round-trip?" question. Patches
   exist only to absorb exponent diversity, never precision loss.

DecimalByteParts applies to the coefficients unchanged: a signed i64 MSP plus u64 lower parts,
compressed independently. A page whose coefficients all fit 64 bits stores a single i64 part.

Scalar

Scalar::DecFloat carrying (coefficient: i128, exponent: i16), with the IEEE special values
(±Infinity, NaN, -0) represented in the coefficient/exponent encoding rather than as separate
variants, matching how the IEEE interchange format does it.

Comparison needs both IEEE orderings, which ANSI SQL already establish as the expected API
surface: numeric comparison (cohort-insensitive, so 2.00 == 2.0) for ==/<, and a separate
total-order predicate for sorting/TOTALORDER semantics.

Why decomposed rather than 16 opaque bytes

FixedSizeBinary(16) holding raw IEEE bytes would be simpler, but gives up three things:

1. Statistics. The BID encoding is not lexicographically ordered by value — the combination field
   puts exponent bits ahead of significand bits, so byte-wise comparison disagrees with numeric order
   across differing exponents. Byte-wise min/max on an opaque column is therefore silently wrong.
2. Compression. Real decimal data has one exponent per page; decomposed, that costs nearly
   nothing, while an opaque blob is incompressible.
3. Compute. Min/max on the exponent answers "does this batch share one exponent?" — the predicate
   that unlocks coefficient-only arithmetic.

Measurements

We benchmarked decimal128 arithmetic across available implementations (i9-13900K, GCC 11.4,
-O3 -march=native, one translation unit so every contender sees bit-identical values and the same
loop shape; 2M values × 5 reps). Slowdown vs native double, money-shaped data:

library	compare	add	multiply
Intel DFP (BID)	41×	54×	67×
IBM decNumber decQuad (DPD)	54×	93×	59×
Boost.Decimal	29×	36×	10×

Against the same operations on raw coefficients with the exponent hoisted to a batch scalar — what
the decomposed layout enables:

kernel	compare	add	multiply
int64 coefficient	4,722 µs	10,486 µs	13,564 µs
best library (Boost.Decimal)	99,335 µs	48,235 µs	85,361 µs

4.6–21× faster than the best scalar library, within 1.3–1.7× of hardware double. int64
coefficient compare is faster than double compare.

From the generated assembly, the coefficient kernels vectorize to packed SIMD when three conditions
hold together: 64-bit coefficients, no in-loop overflow check, and a non-widening multiply. The
layout above is what makes all three knowable from page metadata rather than per value.

The scalar libraries can't reach this because their fast paths are necessarily per-element — the
API is scalar, so the uniformity check is re-paid on every call and can never be hoisted out of a
loop. Storing the decomposition turns it into a page-header read.

Open questions / caveats

Non-normalization is required, and it complicates patches. IEEE decimal128 is deliberately
non-normalized: 2.00 (200E-2) and 2.0 (20E-1) are distinct representations that compare
equal, and the BSON spec requires that round-tripping "MUST not change its value or representation in
any way". So a conforming implementation cannot canonicalize on write — meaning a column of 2.00
and 2.0 carries two exponents despite being numerically uniform. The patch mechanism should stay
cheap rather than assume near-zero patch rates. (Snowflake's DECFLOAT normalizes and consequently
cannot round-trip BSON; worth being deliberate about which side to land on.)

Clamping. The quantum exponent caps below the value range (6111 vs 6144 for decimal128), so
values near the top of the range are stored with the coefficient padded and the exponent reduced.
Lossless, but the encoder has to do it rather than reject.

ExtDType is a viable interim and would work today with a struct storage type, the way uuid
and datetime are handled — but it leaves the encoding on the table, since the compressor can't know
the struct's two fields are an exponent and a coefficient.

BID vs DPD. BSON mandates BID, and Intel DFP (which MongoDB vendors as
src/third_party/IntelRDFPMathLib20U1) is BID-native. A BID coefficient is a plain two's-complement
integer, so it feeds DecimalByteParts with no transcoding. DPD would require conversion per value.

---