Bug
Grouped approx_distinct can panic while DataFusion materializes intermediate HyperLogLog states. In DataFusion 54.1.0, each HLL state contains 16,384 one-byte registers but is represented as ScalarValue::Binary and declared as Arrow DataType::Binary.
GroupsAccumulatorAdapter::state collects one scalar state per group and calls ScalarValue::iter_to_array. Arrow BinaryArray uses signed 32-bit offsets, so this deterministic boundary is unrepresentable:
131,072 groups * 16,384 bytes = 2,147,483,648 bytes
The resulting panic is:
byte array offset overflow
The stack reaches GroupedHashAggregateStream::emit -> GroupsAccumulatorAdapter::state -> ScalarValue::iter_to_array.
Spill does not prevent it
The branch-54 grouped hash aggregate spill path first calls emit(EmitTo::All, true), so all current group states are materialized in one record batch before spill can write them. Terminal output similarly materializes EmitTo::All and only slices the resulting batch afterward.
This means the panic can occur while trying to spill, and memory pressure is not applied before a potentially multi-gigabyte temporary state array is built.
Suggested direction
- Represent dense fixed-width HLL register state without cumulative variable-width offsets, such as
FixedSizeBinary(16384) for the branch-54 implementation.
- Materialize terminal and spill aggregate state in bounded chunks rather than building
EmitTo::All first.
- Propagate memory or spill exhaustion as
DataFusionError::ResourcesExhausted rather than unwinding.
A direct 131,072-group regression requires about 2 GiB per HLL expression, so practical CI coverage can verify the fixed-size state schema and use a small configured emission-byte threshold or synthetic fixed-size accumulator to prove bounded draining.
Related work
Newer DataFusion has evolved its grouped HLL implementation, including sparse state, so the exact state representation on main may need a different design. The invariant remains that dense HLL state and grouped spill/output must not depend on one variable-width Arrow array with 32-bit cumulative offsets.
Bug
Grouped
approx_distinctcan panic while DataFusion materializes intermediate HyperLogLog states. In DataFusion 54.1.0, each HLL state contains 16,384 one-byte registers but is represented asScalarValue::Binaryand declared as ArrowDataType::Binary.GroupsAccumulatorAdapter::statecollects one scalar state per group and callsScalarValue::iter_to_array. ArrowBinaryArrayuses signed 32-bit offsets, so this deterministic boundary is unrepresentable:The resulting panic is:
The stack reaches
GroupedHashAggregateStream::emit -> GroupsAccumulatorAdapter::state -> ScalarValue::iter_to_array.Spill does not prevent it
The branch-54 grouped hash aggregate spill path first calls
emit(EmitTo::All, true), so all current group states are materialized in one record batch before spill can write them. Terminal output similarly materializesEmitTo::Alland only slices the resulting batch afterward.This means the panic can occur while trying to spill, and memory pressure is not applied before a potentially multi-gigabyte temporary state array is built.
Suggested direction
FixedSizeBinary(16384)for the branch-54 implementation.EmitTo::Allfirst.DataFusionError::ResourcesExhaustedrather than unwinding.A direct 131,072-group regression requires about 2 GiB per HLL expression, so practical CI coverage can verify the fixed-size state schema and use a small configured emission-byte threshold or synthetic fixed-size accumulator to prove bounded draining.
Related work
EmitTo::Firstmaintains unused GroupValues lookup state #23178 documents performance concerns around repeatedEmitTo::First.partialandfinalmode aggregation #24061 is related ongoing spill work in newer aggregate streams.Newer DataFusion has evolved its grouped HLL implementation, including sparse state, so the exact state representation on
mainmay need a different design. The invariant remains that dense HLL state and grouped spill/output must not depend on one variable-width Arrow array with 32-bit cumulative offsets.