Repository navigation
ArrowMetal 0.6.0
Everything below is new in 0.6.0. The group-count probe of the Polars engine and of datafusion-arrowmetal
leaves a group-by whose rows are concentrated on a few of its keys to the CPU engine: on Polars 1.44.1 the
default no longer runs the GPU where it is slower on skewed keys (0.5.0: 13 of 72 cases) and picks the faster
engine in 68 of 72 (0.5.0: 59). The differential matrix and the Python suites run on pyarrow 26.0.0, and
remainder, new in Arrow v26, runs on the GPU.
- pyarrow 26.0.0. The differential matrix and the Python test suites run on pyarrow 26.0.0 (CI pin, venv).
Four pyarrow bugs the matrix worked around on 25.0.1 are fixed in 26.0.0 (utf8_normalizecomposing for NFC
and NFKC, by our apache/arrow#51237;pairwise_diff, the boolean fill-null kernels andwinsorizeon a
sliced array honouring the offset): their pinning tests failed on 26.0.0 as designed and were retired with
the workarounds, so the oracle ispyarrow.computeon the slice itself again. Thebinary_slice
default-stop overflow remains (apache/arrow#51222), with its pinning test. Three more pinned behaviours
changed in 26.0.0 and their tests say so now (cumulative_max's default start, the out-of-range
float-to-int cast, the fixed-size-list Parquet file pyarrow's reader used to reject), and the registry
carries 26.0.0's five new function names:remainderruns on the GPU asmodulo()has always computed it
(the C%),remainder_checked, the flooredmodulo/modulo_checkedandsearch_sortedare recorded
as missing. - The group-count probe on keys that are not evenly frequent. The group-count probe of the Polars engine
and ofdatafusion-arrowmetalestimates with Chao1 for sampling without replacement (Chao and Lin 2012) and
takes an estimate only when the sample before it confirms it; a range still unconfirmed at the largest
sample reaches the Good-Turing bound. A group-by whose rows are concentrated on a few of its keys (the
estimate at least 30 times the effective number of keys) is left to the CPU engine whatever its count: on
such keys the GPU group-by measured slower than Polars in 69 of 72 cases
(Benchmarks/results/group_skew_2026-11-05.csv). On Polars 1.44.1 the default picks the faster engine in 68
of 72 skewed cases (was 59) and 72 of 72 with log-normal group sizes (was 65), and no longer runs the GPU
where it is slower (was 13 cases, 0.42x to 0.93x); on 2.0.0, 0 such runs (was 5). Uniform keys get the same
decisions, at 71 to 720 µs a probe.Benchmarks/group_skew_bench.pyand thegskewfamily of
datafusion/examples/bench.rstime it (docs/FINDINGS.md, Round 23).