Support index pruning for IS NOT DISTINCT FROM and IS TRUE#110006
Conversation
KeyCondition::atom_map did not map isNotDistinctFrom (the <=> operator that IS NOT DISTINCT FROM lowers to), so `k IS NOT DISTINCT FROM 42` fell through to Condition: true and scanned all granules. For a non-NULL constant, key <=> c matches the same rows as key = c for both Nullable and non-Nullable keys, so it is mapped to the same FUNCTION_IN_RANGE atom as equals. NULL constants are rejected before the atom is built, because key <=> NULL means "key IS NULL", not "key = NULL". (k = 42) IS TRUE is lowered by the analyzer to isNotDistinctFrom(equals(k, 42), true), where the key-side argument is a boolean sub-expression, not the key column. tryRewriteIsTrueCondition unwraps isNotDistinctFrom(X, true) to X in truth-tested boolean position when X is a boolean-returning function, so the inner predicate becomes a prunable key atom. The rewrite is gated to boolean-result functions because for a non-boolean X, X <=> true means X = 1, not "X is truthy". Closes: ClickHouse#109995 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gate (click to expand)
Session id: cron:clickhouse-worker-slot-3:20260710-140500 |
|
cc @CurtizJ @shankar-iyer for review: this extends |
|
Workflow [PR], commit [8edfda4] Summary: ✅
AI ReviewSummaryThis PR extends Missing context / blind spots
Final Verdict
|
|
@groeneai also check partition pruning in this test case: https://fiddle.clickhouse.com/c6e6ab80-3c5b-4146-8146-ad256fa5fb1d |
The isNotDistinctFrom KeyCondition mapping added in this PR now prunes the granules for the boolean-key IS TRUE / IS FALSE predicates in 04304_primary_key_is_true_false_unknown (SelectedMarks: IS FALSE 3->1, IS TRUE 3->2). Results are unchanged; only granule reads drop, which is the intended optimization. Update the reference and the explanatory comment accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tryRewriteIsTrueCondition gated on the raw child of isNotDistinctFrom(X, true) being a boolean-result function. Equivalent wrapped forms like CAST(k = 42, 'UInt8') IS TRUE or materialize(k = 42) IS TRUE reached the gate as CAST / materialize (not boolean-result functions) and returned nullptr, so the wrapped predicate did not become a prunable key atom even though the later clone strips those wrappers anyway. Peel alias / materialize / trivial CAST (the same wrappers cloneDAGWithInversionPushDown strips) before the gate so these equivalent predicates prune identically. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Fixed in cb61eef. Verified on a
Result counts are unchanged (1 row each). The gate checks the boolean-result-ness of the peeled node but still clones the original predicate, so the recursion strips the wrappers consistently under boolean context. Regression coverage added to Note: a non-trivial |
Pre-PR validation gate (bot follow-up: wrapper peeling)
Session id: cron:clickhouse-worker-slot-4:20260710-162000 |
Partition pruning goes through a separate KeyCondition path (PartitionPruner over the partition value / minmax) than primary-key granule pruning. The isNotDistinctFrom atom map and IS TRUE rewrite already apply there, so b IS TRUE and b IS NOT DISTINCT FROM true now prune partitions the same as b = true. Add EXPLAIN indexes=1 Parts-count assertions plus correctness checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Partition pruning is covered by this PR too. The fiddle scenario reproduced on the patched binary (
Added regression coverage to |
…true) Follow-up to the IS TRUE / IS NOT DISTINCT FROM rewrite. The same KeyCondition rewrite now also peels two more truth-equivalent positive boolean wrappers so a wrapped predicate becomes a prunable key atom: - notEquals(X, false) i.e. (k = 42) != false - in(X, <all-true const set>) i.e. (k = 42) IN (true) Both are truth-equivalent to bare X only when X is boolean-valued, so the same boolean-result gate as the IS TRUE path applies (extracted into predicateIsBooleanResult). The IN path only handles literal/constant sets whose explicit elements are available and every element is exactly true (numeric 1); it declines otherwise (X IN (false), X IN (true, false), X IN (2), empty set, subquery set), falling back to existing behavior. Benefits every index that goes through this shared canonicalizer: primary key, minmax, bloom_filter and partition pruning. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Extended the same KeyCondition rewrite to two more truth-equivalent positive boolean wrappers (d0bfa20), so they prune identically to the bare atom across primary key, minmax, bloom_filter and partition pruning:
Refactored the boolean-result gate into Regression coverage added to Pre-PR validation gate (follow-up: != false and IN (only-true))
Session id: cron:clickhouse-worker-slot-15:20260710-210200 |
Build (arm_tidy) failed with cppcoreguidelines-init-variables: 'variable expected_const is not initialized'. Every non-return path assigns it, so the initializer is defensive/dead, but tidy runs as WERROR. Initialize to 0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tryRewriteInTruthyCondition called buildOrderedSetInplace unconditionally while cloning the key-condition DAG, which runs for every query on the primary-key path. For X IN (SELECT ...) this executed the subquery purely for analysis, so EXPLAIN SELECT ... WHERE c IN (SELECT throwIf(1)) with use_skip_indexes=0 threw (server abort in debug/sanitizer builds) even though no index is used, breaking 02707_skip_index_with_in. Only inspect an already-built set (FutureSet::get() non-null for literal tuple and storage sets); decline a not-yet-run subquery set. This matches the intent that the IN set is not built when indexes are not used. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…onotonic chain applyFunction (the range path) passed a raw ColumnLowCardinality to func->execute while the monotonic-function chain is built against the outer-LowCardinality-stripped key type, so a specialized UInt8->Bool CAST wrapper did checkAndGetColumn<ColumnUInt8> on a ColumnLowCardinality and aborted with a bad cast. Strip outer LowCardinality from the argument column and type (and keep the cached result full), matching the sibling applyFunctionChainToColumn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Pushed d05c97b to fix an AST-fuzzer finding on this PR ( Root cause is pre-existing on master, not introduced by this PR: Regression coverage added to |
…table The lc_bool regression table sets index_granularity_bytes = 0 (non-adaptive granularity), which is incompatible with the default min_bytes_for_wide_part (10 MB Compact threshold). MergeTree then logs a Warning to stderr, and Fast test fails any test that writes to stderr. Add min_bytes_for_wide_part = 0 to force Wide format, matching the non-adaptive granularity. Test output is unchanged (reference already matched); this only silences the warning. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI finish ledger — 39b6aceEvery failure below has an owner (a fixing PR). Only
Session id: cron:our-pr-ci-monitor:20260711-053000 |
…L to isNull Address two clickhouse-gh[bot] review Majors on the positive-boolean-wrapper key pruning: - Derive the boolean-result gate in predicateIsBooleanResult from KeyCondition::atom_map (single source of truth) instead of a hand-maintained comparison list, so IS TRUE / != false / IN (true) wrappers peel to ANY prunable boolean atom KeyCondition supports (has, startsWith, startsWithUTF8, match, pointInPolygon, ...), not just comparisons. startsWith(s,'ab') IS TRUE now prunes the key the same as bare startsWith(s,'ab'). - Route the NULL arm of isNotDistinctFrom onto the existing isNull atom: key <=> NULL is equivalent to isNull(key), so a Nullable PK / minmax index now prunes to the NULL granule exactly instead of scanning every granule. Add 04401 coverage for the newly-covered forms and flip the previously-weaker IS NOT DISTINCT FROM NULL assertion to assert pruning. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Both addressed in 6be0f9e. Major 1 (IS TRUE gate too narrow). Major 2 (NULL arm of IS NOT DISTINCT FROM). The Both directions A/B verified on Build-ID-matched binaries. 04401 extended ( |
Pre-PR validation gate (click to expand) — commit 6be0f9e
Session id: cron:clickhouse-worker-slot-1:20260711-054900 |
ifNull(k = 42, 0) IS TRUE (and the != false / IN (true) forms) lowered to isNotDistinctFrom(ifNull(equals(k, 42), 0), true) did not prune: the boolean-result gate in predicateIsBooleanResult rejects the ifNull wrapper, so tryRewriteIsTrueCondition declines and the fallback clone with boolean_context = false denies the existing coalesce rewrite its chance too, leaving Condition: true even though bare ifNull(k = 42, 0) already prunes when allow_key_condition_coalesce_rewrite is enabled. Recurse the boolean-result gate through a falsy-zero ifNull/coalesce wrapper (gated by the same setting): ifNull(Y, 0) is boolean-valued iff Y is, so the outer peel enters boolean context and the existing recursion unwraps the ifNull to the prunable inner atom. Factor the ifNull/coalesce(X, 0) shape recognition into isFalsyZeroCoalesceCondition, shared by the gate and tryRewriteCoalesceCondition. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
functionIsTotalOnKeyDomain used canBeSafelyCast + is_always_monotonic to decide totality. That misses intDiv(k, -1): getMonotonicityForRange reports is_always_monotonic for a -1 divisor, yet intDiv(Int_MIN, -1) overflows and raises ILLEGAL_DIVISION for key widths without the constant-divisor specialization (Int8/Int16/Int128). So isNull(intDiv(k, -1)) could reuse the bare isNull atom and prune the non-NULL granule, suppressing the exception. Replace the heuristics with a behavioral boundary probe: run the wrapper on the extreme values of its integer key domain via the same execute path a scan uses, and decline the isNull-atom reuse if it raises. This catches conversion overflow, div-by-zero and the signed min/-1 case uniformly, and admits the Int32/Int64 intDiv(k, -1) forms that wrap (never raise). Non-integer domains fall back to the conservative type-level canBeSafelyCast check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gate (click to expand)
Session id: cron:clickhouse-worker-slot-4:20260712-121200 |
| if (!key_arg_type || !isNullableOrLowCardinalityNullable(key_arg_type)) | ||
| return true; | ||
|
|
||
| try |
There was a problem hiding this comment.
These two helpers only gate when isNull / IS NOT DISTINCT FROM NULL over a monotonic-wrapped key may reuse the bare isNull atom. I can drop the runtime probing entirely and reuse that atom only for a bare Nullable key (empty function chain), falling back to a full scan whenever the key is wrapped by any monotonic function. Core IS TRUE / IS NOT DISTINCT FROM <non-null> pruning is unaffected. Want me to go that way, or would you prefer the wrapped-key null atom handled differently?
There was a problem hiding this comment.
Done in 3d34f3c. Removed both runtime-probing helpers. The isNull atom is now reused only for a bare Nullable key (empty monotonic-functions chain); any wrapper declines and full-scans, which is always correct. This also subsumes every wrapper class the earlier rounds chased (NULL-erasing, overflow-checked conversions, intDiv). Core IS TRUE / IS NOT DISTINCT FROM pruning is unchanged.
| auto column = from_type->createColumn(); | ||
| for (const auto & field : *extremes) | ||
| column->insert(field); | ||
| try |
There was a problem hiding this comment.
Removed in 3d34f3c along with the NULL-preservation probe; the totality machinery is gone. A wrapped key no longer reuses the isNull atom at all.
The isNull/isNotNull atoms (and the key IS NOT DISTINCT FROM NULL branch that reuses them) ignore the key monotonic-functions chain, so they are sound only for a bare Nullable key. A monotonic wrapper can erase NULL (ifNull, coalesce, CAST to non-Nullable) or be partial (toDateTime(Date32) / intDiv(k, 0) throw on some rows), which would let isNull(wrapper(k)) narrow to the NULL granule and mark it exact-true, returning wrong exact-count results. Decline the atom for any wrapped key and fall back to a full scan, which is always correct. This removes the runtime function-probing helpers (functionPreservesNulls, functionIsTotalOnKeyDomain) in favor of a plain empty-chain check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gate (click to expand)
Session id: cron:clickhouse-worker-slot-0:20260712-165800 |
The 04304 EXPLAIN/SelectedMarks assertions hard-code granule counts, but bool_pk and int_pk left index_granularity_bytes randomizable, so a small randomized adaptive-granularity value could split the tables into a different number of marks and break the assertions. Pin index_granularity_bytes = 0 and min_bytes_for_wide_part = 0 on both tables, matching 04401_date32_zerotransform_monotonicity_pruning in this PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gate (click to expand)
Verified via Session id: cron:clickhouse-worker-slot-3:20260712-181200 |
CI finish ledger — d551f23Every failure below has an owner (a fixing PR). No failure is PR-caused; the PR's KeyCondition/04304/04401 changes are unrelated to the two flaky checks that failed.
Session id: cron:our-pr-ci-monitor:20260712-230000 |
LLVM Coverage Report
Changed lines: Changed C/C++ lines covered: 139/155 (89.68%) · Uncovered code |
…ey-over-merge Reconcile the KeyCondition::applyFunction cache-miss LowCardinality strip with the sibling fix from ClickHouse#110006 (merged): keep the recursive strip (covers nested Array(LowCardinality(T))) for the input column and type, and keep ClickHouse#110006's result hardening (removeLowCardinality on the result type + convertToFullColumnIfLowCardinality on the result column). applyFunctionForField and the sparse checkInHyperrectangle caller retain their recursive strips.
Cherry pick #110006 to 26.6: Support index pruning for IS NOT DISTINCT FROM and IS TRUE
Backport #110006 to 26.6: Support index pruning for IS NOT DISTINCT FROM and IS TRUE
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
IS NOT DISTINCT FROMandIS TRUEnow use primary-key and minmax indexes to prune granules, the same as=. Previouslyk IS NOT DISTINCT FROM 42and(k = 42) IS TRUEscanned all granules.Description
Closes #109995.
KeyCondition::atom_mapdid not mapisNotDistinctFrom(the<=>operator thatIS NOT DISTINCT FROMlowers to), sok IS NOT DISTINCT FROM 42fell through toCondition: trueand scanned all granules. For a non-NULL constant,key <=> cmatches the same rows askey = cfor both Nullable and non-Nullable keys, so it is mapped to the sameFUNCTION_IN_RANGEatom asequals. NULL constants are rejected before the atom is built, becausekey <=> NULLmeans "key IS NULL", not "key = NULL".(k = 42) IS TRUEis lowered by the analyzer toisNotDistinctFrom(equals(k, 42), true), where the key-side argument is a boolean sub-expression, not the key column.tryRewriteIsTrueConditionunwrapsisNotDistinctFrom(X, true)toXin truth-tested boolean position whenXis a boolean-returning function, so the inner predicate becomes a prunable key atom. The rewrite is gated to boolean-result functions because for a non-booleanX,X <=> truemeansX = 1, not "X is truthy".Version info
26.7.1.1199(included in26.7and later)26.6.2.100