fix: skip count-distinct byte tests under force_hash_collisions - #25020
Merged
kosiew merged 1 commit intoSep 7, 2026
Merged
Conversation
`ungrouped_utf8_accumulator_is_never_worse_than_a_pre_allocated_set`
and its `Utf8View` counterpart in
`datafusion/functions-aggregate-common/src/aggregate/count_distinct/bytes.rs`
insert up to 500,000 distinct values into `ArrowBytesSet`/`ArrowBytesViewSet`
twice per cardinality (once lazily built, once pre-allocated). Under normal
hashing this is O(n) per insert. Under `force_hash_collisions` every value
hashes to the same bucket, degrading the set to a linear scan per insert,
i.e. O(n^2) overall.
Empirically measured locally with a throwaway timing probe (since removed):
n=100 -> 297us
n=500 -> 3.7ms
n=1000 -> 14ms
n=2000 -> 54ms
n=5000 -> 347ms
which is consistent with the quadratic growth reported in apache#25011 (some CI
runs completing in ~2h39m/4h31m for the two tests, others exceeding the
360-minute job limit and getting cancelled).
This mirrors the existing `force_hash_collisions` precedent for exactly
this class of problem: `count_distinct_spill` in
`datafusion/core/tests/memory_limit/mod.rs` (added in apache#24918) is gated with
`#[cfg(not(feature = "force_hash_collisions"))]` because its assertions
depend on a real hash distribution across partitions. The same reasoning
applies here — these tests assert on allocator sizes that only make sense
under real hashing, and forcing every key into one bucket does not exercise
any behavior the test is meant to protect, it only inflates the runtime.
Changes:
- `datafusion/functions-aggregate-common/Cargo.toml`: declare a local
`force_hash_collisions` feature forwarding to
`datafusion-common/force_hash_collisions`, matching the same forwarding
pattern used in `datafusion/core/Cargo.toml`. Needed because Cargo does
not propagate a dependency's active feature into a consuming crate's own
`cfg(feature = ...)` checks - the crate must declare (and forward) the
feature itself for its own `#[cfg(feature = "force_hash_collisions")]` to
respond to the workspace-level `--features force_hash_collisions` flag
the affected CI job passes.
- `bytes.rs`: gate the whole `mod tests` block with
`#[cfg(all(test, not(feature = "force_hash_collisions")))]`, since it
contains only these two tests and their shared helpers.
No production code changes; no reduction in cardinality or coverage under
normal (non-collision-forced) test runs, where both tests still run exactly
as before across all 7 cardinalities up to 500,000.
Verified:
- `cargo test -p datafusion-functions-aggregate-common --lib -- count_distinct::bytes`
(feature off): both tests still run and pass, 0.75s.
- `cargo test -p datafusion-functions-aggregate-common --lib --features force_hash_collisions -- count_distinct::bytes`
(crate-local feature on): 0 tests run, clean compile.
- The exact affected CI job command, `cargo test --profile ci --exclude
datafusion-examples --exclude datafusion-benchmarks --exclude
datafusion-sqllogictest --exclude datafusion-cli --workspace --lib --tests
--features=force_hash_collisions,avro`: the crate's test binary reports
47 tests (49 minus the 2 gated ones), all passing, with neither
`ungrouped_utf8_accumulator_is_never_worse_than_a_pre_allocated_set` nor
its view counterpart appearing in the run.
- `cargo fmt --check` and the exact `ci/scripts/rust_clippy.sh`
(`cargo clippy --all-targets --workspace --features
avro,integration-tests,extended_tests -- -D warnings`): both clean.
Closes apache#25011
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #25020 +/- ##
==========================================
- Coverage 81.71% 81.71% -0.01%
==========================================
Files 1127 1127
Lines 416051 416051
Branches 416051 416051
==========================================
- Hits 339974 339962 -12
- Misses 56091 56097 +6
- Partials 19986 19992 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
adriangb
approved these changes
Sep 7, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The
cargo test hash collisions (amd64)CI job hangs for hours (sometimes hitting the 360-minute job limit and getting cancelled) in two tests indatafusion/functions-aggregate-common/src/aggregate/count_distinct/bytes.rs:Root cause: both tests insert up to 500,000 distinct values into
ArrowBytesSet/ArrowBytesViewSet, twice per cardinality inCARDINALITIES(once into a lazily-constructed set, once into a pre-allocated one), to compare their reported.size(). Under normal hashing this is O(n) per insert. Underforce_hash_collisions(datafusion/common/src/hash_utils.rs#L1184-L1195) every value hashes to the same bucket, so the underlying hash table degrades to a linear scan per insert - O(n^2) overall for a set built up to n elements.I confirmed this is quadratic, not just slow, with a throwaway local timing probe over the same insert pattern under
--features datafusion-common/force_hash_collisions(removed before this PR, shown here for reference):Each 2x step in n is roughly a 4x step in time, consistent with O(n^2), and consistent with the multi-hour runtimes reported in #25011 for n up to 500,000.
On the question raised in #25011 ("what behavior or regression boundaries are the 100,000 and 500,000 cardinalities intended to protect, and what approach would preserve that coverage?"): the assertions in
assert_lazy_is_not_worsecompare allocator sizes reported by a real hash-table implementation against a pre-allocated one, at cardinalities chosen to span both sides of the warm-up capacity (PER_GROUP_SCALE) and the point where the two constructors converge (UNGROUPED_SCALE). None of that is about hash collision behavior - forcing every key into one bucket doesn't exercise a code path these tests are meant to protect, it just makes every insert scan the one bucket's full contents, which is why the cost goes quadratic without adding coverage.This is the same situation the
force_hash_collisionsfeature already has an established answer for:count_distinct_spillindatafusion/core/tests/memory_limit/mod.rs(added in #24918) is gated with#[cfg(not(feature = "force_hash_collisions"))]because its assertions depend on a real hash distribution across partitions and don't mean anything under forced collisions. This PR applies the identical pattern here, rather than reducing cardinality or otherwise changing what real-hashing runs cover.What changes are included in this PR?
datafusion/functions-aggregate-common/Cargo.toml: add a[features]section declaringforce_hash_collisions = ["datafusion-common/force_hash_collisions"], forwarding todatafusion-common's feature of the same name. This crate previously declared no features of its own. Cargo does not propagate a dependency's active feature into a consuming crate's owncfg(feature = ...)checks, so without this forwarding declaration, a#[cfg(feature = "force_hash_collisions")]inside this crate would never see the workspace-level--features force_hash_collisionsflag the affected CI job passes (cargo test --workspace --features=force_hash_collisions,avro). This mirrors the exact forwarding pattern already used indatafusion/core/Cargo.tomlfor the same feature name.datafusion/functions-aggregate-common/src/aggregate/count_distinct/bytes.rs: gate the wholemod testsblock (it contains only these two tests and their shared helpers - nothing else needs to stay compiled either way) with#[cfg(all(test, not(feature = "force_hash_collisions")))], with a doc comment explaining the O(n^2) mechanism and linking back to this issue.No production code changes. No reduction in cardinality or coverage for the normal (non-collision-forced) test run - both tests still run exactly as before, across all 7 cardinalities up to 500,000, whenever
force_hash_collisionsis off.What is the testing strategy for this PR?
This is a test-only change, verified by running the tests both ways:
cargo test -p datafusion-functions-aggregate-common --lib -- count_distinct::bytesstill runs and passes both tests in ~0.8s.cargo test -p datafusion-functions-aggregate-common --lib --features force_hash_collisions -- count_distinct::bytesruns 0 tests with a clean compile - confirming the gate compiles out cleanly rather than silently failing to match.cd datafusion && cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-sqllogictest --exclude datafusion-cli --workspace --lib --tests --features=force_hash_collisions,avro): thedatafusion-functions-aggregate-commontest binary reports 47 tests (49 minus the 2 gated ones) all passing, with neitherungrouped_utf8_accumulator_is_never_worse_than_a_pre_allocated_setnor itsUtf8Viewcounterpart appearing anywhere in the run - confirming the workspace-level feature flag correctly reaches the new local feature via Cargo's feature unification, not just the crate-local invocation.cargo fmt --checkand the exact CI clippy invocation (ci/scripts/rust_clippy.sh, i.e.cargo clippy --all-targets --workspace --features avro,integration-tests,extended_tests -- -D warnings) both pass clean across the whole workspace.Are there any user-facing changes?
None. This only changes which tests compile under a testing-only feature flag; there is no change to any public API or runtime behavior.