Support tuple lexicographic comparison for index analysis - #108290
Support tuple lexicographic comparison for index analysis#108290fastio wants to merge 9 commits into
Conversation
Let KeyCondition use (k1, k2, ...) <op> (c1, c2, ...) for primary key and minmax granule pruning, gated by analyze_index_with_tuple_lexicographic_comparison.
nihalzp
added
the
can be tested
|
Workflow [PR], commit [1e6415e] Summary: ❌
AI ReviewSummaryThis PR teaches Findings❌ Blockers
Final VerdictChanges requested. |
clickhouse-gh
Bot
added
the
pr-improvement
![clickhouse-gh[bot]](https://avatars.githubusercontent.com/in/928874?s=60&v=4){kind=small}
The missing terminator swallowed the next DECLARE, breaking the link; also record the history entry as false->true for compatibility.
Add the missing branch to the sparse checkInHyperrectangle overload, factoring the corner comparison into a shared helper to avoid an exception.
|
Dear @nihalzp, you haven't been active on this PR for 30 days. You will be unassigned. Will you continue working on it? If so, please feel free to reassign yourself. |
Record `analyze_index_with_tuple_lexicographic_comparison` under the current settings-history version and pin the tuple-index test to legacy `EXPLAIN` output. CI: ClickHouse#108290
| const bool included = use_lower ? r.left_included : r.right_included; | ||
| const Field & ci = c[i]; | ||
|
|
||
| if (isRealNull(bound) || isRealNull(ci) || bound.isNaN() || ci.isNaN()) |
There was a problem hiding this comment.
compareTupleCorner only treats a literal Field::Null as indeterminate, but primary-key bounds never arrive here that way: MergeTreeDataSelectExecutor::create_field_ref rewrites nullable key values to POSITIVE_INFINITY before checkInRange runs. On a mark whose only tuple is (4, NULL), this code therefore reports (a, b) >= (4, 3) as definitely true (lower == upper == Greater), even though tuple comparison with a later NULL is NULL and should not match. That makes the new tuple atom non-conservative on nullable keys and gives an unsound can_be_false to exact-range users. Please either treat encoded-NULL bounds as indeterminate here or relax / disable the tuple atom whenever a later compared key column can still be NULL.
| out.function = RPNElement::FUNCTION_TUPLE_LEXICOGRAPHIC; | ||
| out.key_columns = std::move(mapped_key_columns); | ||
| out.tuple_lexicographic_constant = std::move(mapped_constant); | ||
| out.tuple_lexicographic_chains = std::move(mapped_chains); |
There was a problem hiding this comment.
These new per-element monotonic chains never make KeyCondition::hasMonotonicFunctionsChain() return true, because that helper still only checks element.monotonic_functions_chain. The result is that MergeTreeDataSelectExecutor::create_field_ref() takes the explicit-Field path for tuple predicates, so applyMonotonicFunctionsChainToRange() executes functions like toDate separately for every boundary instead of using the cached block-backed FieldRef path that scalar predicates get. The new functional-tuple case regresses PK analysis exactly on the hot path this optimization adds; hasMonotonicFunctionsChain() needs to look at tuple_lexicographic_chains too.
| if (i < chains.size() && !chains[i].empty()) | ||
| { | ||
| std::optional<Range> new_range | ||
| = applyMonotonicFunctionsChainToRange(range, chains[i], data_types[kc], single_point); |
There was a problem hiding this comment.
The scalar FUNCTION_IN_RANGE path strips LowCardinality before calling applyMonotonicFunctionsChainToRange() (recursiveRemoveLowCardinality(data_types[key_column]) a few lines above), but the tuple branch feeds the raw data_types[kc] / sparse_data_types[sparse_pos] back into the same helper. That reintroduces the mismatch the scalar path already had to harden against: the chain is built against the nested type, so a tuple predicate like (CAST(lc_key AS Int32), id) < (...) can hit the LowCardinality bad-cast path here instead of behaving like the scalar equivalent. This is reachable today because hasMonotonicFunctionsChain() still ignores tuple_lexicographic_chains, so create_field_ref() materializes explicit bounds for tuple-chain predicates. Please strip LowCardinality here (and in the sparse branch below) to match the scalar path.
LLVM Coverage Report
Changed lines: Changed C/C++ lines covered: 259/283 (91.52%) · Uncovered code |
3 participants
Issue #75086.
Use lexicographic tuple comparisons on key columns, e.g.
(x, y) < (1, 5), for granule pruning. Previously such conditions were dropped toFUNCTION_UNKNOWNand read all granules, whilex < 1 OR (x = 1 AND y < 5)and(x, y) IN (...)worked.Gated by the new setting
analyze_index_with_tuple_lexicographic_comparison(defaulttrue). Benefits primary key, part-level minmax, and minmax skip indexes.Example:
Before:
See: https://fiddle.clickhouse.com/46ac4c2d-eb71-4bd7-9ea3-0fa61a5c83fc
After:
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Use lexicographic tuple comparisons on key columns, e.g.
(a, b) < (1, 5), for primary key and minmax index granule pruning. Controlled by the new settinganalyze_index_with_tuple_lexicographic_comparison(enabled by default).Documentation entry for user-facing changes
src/Core/Settings.cpp)