Combine I/O cost with selectivity in PREWHERE condition ordering - #110695
Combine I/O cost with selectivity in PREWHERE condition ordering#110695Avogar wants to merge 5 commits into
Conversation
When `use_statistics=1` (the default since `auto_statistics_types` was introduced), the PREWHERE optimizer sorted conditions by `estimated_row_count` alone, with `columns_size` only as a tiebreaker. This caused expensive conditions (e.g. Map column, ~500KB) to be placed before cheap ones (e.g. scalar column, ~1KB) whenever the expensive condition appeared more selective — ignoring the I/O cost difference. Apply the classic conjunctive filter ordering rule: sort by `cost / (1 - selectivity)`, i.e. the I/O cost per rejected row. This is computed as `columns_size / max(1, total_rows - estimated_row_count)` and replaces the separate `estimated_row_count, columns_size` pair in the condition comparison tuple. When statistics are unavailable (`estimated_row_count=0`, `total_rows=0`), the formula degrades to `columns_size`, preserving the existing behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Workflow [PR], commit [3344dbe] Summary: ❌
|
clickhouse-gh
Bot
added
the
pr-performance
| @@ -494,6 +497,12 @@ void MergeTreeWhereOptimizer::analyzeImpl(Conditions & res, const RPNBuilderTree | |||
| pk_positions.emplace(cond.min_position_in_primary_key); | |||
| } | |||
|
|
|||
| /// Combine I/O cost with selectivity using the classic conjunctive filter ordering rule: | |||
| /// sort by cost / (1 - selectivity), i.e. cost per rejected row. | |||
There was a problem hiding this comment.
Sorry, I didn't understand why we order by cost per rejected row. Any reference?
There was a problem hiding this comment.
This is what Claude told me, and I trusted this (but this is why I asked you to review as well):
This is the classic rank-ordering theorem for conjunctive predicates (sometimes called the KBZ ordering).
Consider two filters A and B with I/O costs c_A, c_B and selectivities s_A, s_B (fraction of rows that pass):
- A then B: total cost =
c_A + s_A * c_B(always payc_A; payc_Bonly for thes_Afraction that passes A) - B then A: total cost =
c_B + s_B * c_A
A-first is cheaper when:
c_A + s_A * c_B < c_B + s_B * c_A
c_A (1 - s_B) < c_B (1 - s_A)
c_A / (1 - s_A) < c_B / (1 - s_B)
So the optimal order is: sort by c_i / (1 - s_i) ascending -- i.e., cost per rejected row. This generalizes to any number of independent conjuncts.
In the code, columns_size / max(1, total_rows - estimated_row_count) is proportional to c / (1 - s) since total_rows - estimated_row_count = total_rows * (1 - s), and the total_rows denominator is a constant across all conditions that doesn't affect the sort order.
References:
- Ibaraki & Kameda, "On the Optimal Nesting Order for Computing N-Relational Joins", ACM TODS 1984
- Krishnamurthy, Boral & Zaniolo, "Optimization of Nonrecursive Queries", VLDB 1986
- Hellerstein & Stonebraker, "Predicate Migration: Optimizing Queries with Expensive Predicates", SIGMOD 1993
There was a problem hiding this comment.
That makes sense. The fast test is flaky because sometimes the column_size is zero. I asked AI and got reply
Compact parts intentionally don't track per-column sizes.
If that's true, maybe we should compare selectivity when column_size is 0.
|
Tests with statistics failed, need to investigate. Maybe this approach doesn't work |
…re-cost-with-selectivity
…ilable The new `cost_with_selectivity` score is `columns_size / max(1, total_rows - estimated_row_count)`. On compact parts per-column compressed sizes are not tracked, so `columns_size` is 0 and the score collapses to 0 for every condition. Since this score is the only selectivity-bearing key in the sort tuple, all conditions tie and the original WHERE order is kept, discarding the selectivity estimate entirely. Fall back to `estimated_row_count` when `columns_size == 0` so ordering by selectivity is preserved. On wide parts (sizes known) the cost formula is unchanged. This fixes the `02864_statistics_usage` and `04266_statistics_basic_prewhere` fast-test failures. Adds `04811_prewhere_statistics_ordering_compact_parts` to cover the compact-part case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![clickhouse-gh[bot]](https://avatars.githubusercontent.com/in/928874?s=60&v=4){kind=small}
| cond.cost_with_selectivity = static_cast<double>(cond.estimated_row_count); | ||
| else | ||
| cond.cost_with_selectivity = static_cast<double>(cond.columns_size) | ||
| / std::max(1.0, static_cast<double>(total_rows) - static_cast<double>(cond.estimated_row_count)); |
There was a problem hiding this comment.
The cost / (1 - selectivity) rule only works if a predicate that rejects zero rows sorts last. Here max(1, total_rows - estimated_row_count) gives that case a finite score equal to columns_size, so a cheap tautology can still be scheduled before a genuinely selective expensive predicate. That is a regression from the old estimated_row_count ordering, which always pushed estimated_row_count == total_rows to the end.
I think this needs an explicit "rejects 0 rows => sort last" sentinel (for example std::numeric_limits<double>::infinity()) plus a regression that combines estimated_row_count == total_rows with very different columns_size values.
|
📊 Cloud Performance Report ✅ AI verdict: no significant changes detected. K_source=6 K_base=30 flagged=0/65 clickbench🟢 No significant changes tpch_adapted_1_official🟢 No significant changes Debug info
|
The cost-based PREWHERE ordering score divides by columns_size (compressed column size), so the condition order depends on part type and serialization - settings CI randomizes. This made several prewhere-ordering tests flaky. Pin the selectivity-oriented tests (04266, 04304, 03580, 04053) to compact parts, where per-column sizes are 0 and ordering falls back to selectivity alone, independent of size-affecting randomization. For 04513 (which must stay wide to exercise the cost path) split `modality` 50/50 so the reject-count ratio is ~2x while the Map column dwarfs the scalar, keeping cheap-filter-first stable. Verified with clickhouse-test --test-runs 100 -j 10: all affected tests pass 100/100 under randomized settings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build profile diff (arm_release)No arm_release build profile data for commit 3344dbe - the build was skipped, reused from cache, or predates profile upload. |
The cost-per-rejected-row score used max(1, total_rows - estimated_row_count) as the denominator, so a predicate that rejects no rows (estimated_row_count == total_rows) got a finite score equal to columns_size. A cheap tautology could then be scheduled before a genuinely selective but expensive predicate, which regresses the old estimated_row_count ordering. Give such predicates an infinite score so they always sort last, and keep the explicit no-statistics (total_rows == 0) and compact-part (columns_size == 0) fallbacks. Adds 04812_prewhere_statistics_tautology_last. Addresses review comment: #110695 (comment) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2 participants
When
use_statistics=1(the default sinceauto_statistics_typeswas introduced), the PREWHERE optimizer sorted conditions byestimated_row_countalone, withcolumns_sizeonly as a tiebreaker. This caused expensive conditions (e.g. Map column, ~500KB) to be placed before cheap ones (e.g. scalar column, ~1KB) whenever the expensive condition appeared more selective — ignoring the I/O cost difference.Apply the classic conjunctive filter ordering rule: sort by
cost / (1 - selectivity), i.e. the I/O cost per rejected row. This is computed ascolumns_size / max(1, total_rows - estimated_row_count)and replaces the separateestimated_row_count, columns_sizepair in the condition comparison tuple.When statistics are unavailable (
estimated_row_count=0,total_rows=0), the formula degrades tocolumns_size, preserving the existing behavior.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Combine I/O cost with selectivity in PREWHERE condition ordering. It fixes performance regression in PREWHERE execution in some cases introduced after #101275. Part of #110462