Read a row group's bloom filters only when a predicate reaches them (#310) - #315
Merged
Conversation
A bloom filter is consulted only for an equality predicate whose zone map did
not already rule the group out. The reader loaded every candidate group's bloom
filters before evaluating any predicate, so a scan that skipped 19 of 20 groups
paid for the bloom filters of all 20 and reached none of them.
The cost is not incidental. A bloom filter holds one bitmap per column sized by
the group's distinct values, so the eager read scaled with both the column count
and the group size.
Measured with twelve columns, comparing a single group scanned whole against 20
groups with a predicate that keeps exactly one, so the difference over 19 skips
is the per-skip cost:
rows/group baseline probe per-skip
2,000 180 1030 44.7
10,000 228 1468 65.3
50,000 336 3177 149.5
200,000 715 9577 466.4
Per-skip cost is not flat; it grows with the size of the groups being skipped.
Deleting the ColumnarReadBloomList call from the skip path takes 466.4 to 64.5
at 200,000 rows per group, so it is 86% of that cost.
Loading the filters on first use inside the predicate loop takes per-skip to
64.8, matching the deleted-read figure, and the whole query from 9577 buffers to
1946. Results do not change; the filter was always a pruning step.
Adding vector_index = -1 as a scan key to ColumnarReadZoneMapList was tried and
is not the cost: per-skip went 466.4 to 427.9.
test/bloom_lazy.sh, 14 checks. It measures block reads against pgcolumnar.bloom
rather than a total buffer count, so what it asserts is what changed: bloom
reads follow the groups a scan keeps, not the groups it examines. A scan that
keeps every group must read more, so a change that stopped reading filters
cannot pass. Six parity checks against a heap mirror, plus two with
pgcolumnar.enable_bloom_filter off.
Removal proof: restoring the eager read takes bloom blocks from 1 and 1 to 6 and
33 across tables of 6 and 30 groups, turning both property checks red.
This does not close #310. Its 304,233 buffers are at 100M rows and 667 groups;
these probes are at 20 groups.
Green on the five-major matrix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
Collaborator
Author
|
This overlaps #313 (also open, same #310 bloom over-read). They agree on the dominant fix — defer the bloom read until a predicate reaches it, so a group the zone map skips reads no blooms. But #313 goes one step further and I think it should be the one that lands:
Your measurement here (9577 → 1946 buffers) and the CHANGELOG entry are good — I'll fold the CHANGELOG note into #313. Closing this as superseded by #313; reopen if you see a case #313 misses. (Heads-up to @jdatcmd: these two are the same fix from two of my sessions — please review #313, not both.) |
This was referenced Aug 1, 2026
ChronicallyJD
added a commit
to ChronicallyJD/pgcolumnar
that referenced
this pull request
Aug 1, 2026
…pt#310) The entry for commandprompt#310 described the mechanism but measured it only on a 20-group synthetic shape, in buffer counts. That understates the effect and does not name its cause. Adds the figures from the profiled 100M TSBS-cpu run: a single bloom filter is 256 kB, the whole catalog is 3.5 GB, larger than the data it describes, and about 55 percent of the query's CPU sat in anonymous-page faults under the group-skip check from copying it per scan. The clustered hostname query falls from 4610 ms to 106 ms, a factor of 43, against the 30.6x measured on synthetic data. Measurement and profiling by ChronicallyJD in commandprompt#313. The code that PR proposed is already in main via commandprompt#315 and commandprompt#317, and is the same change; its evidence was better than what the merged commits carried, so this keeps it. Refs commandprompt#310, commandprompt#313, commandprompt#314. No issue is closed by this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8 Co-authored-by: ChronicallyJD <ChronicallyJD@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the dominant term in #310. Does not close it: see "What this does not
settle" below.
A bloom filter is consulted only for an equality predicate whose zone map did not
already rule the group out. The reader loaded every candidate group's bloom
filters before evaluating any predicate, so a scan that skipped 19 of 20 groups
paid for the bloom filters of all 20 and reached none of them.
The measurement
For each group size, two tables of identical group size: one group scanned whole
(the data cost of reading one group), and 20 groups with a predicate that keeps
exactly one. The difference over 19 skips is the per-skip cost. Twelve columns.
Per-skip cost is not flat. It grows with the size of the groups being skipped,
because a bloom filter holds one bitmap per column sized by the group's distinct
values.
Deleting the
ColumnarReadBloomListcall from the skip path isolates it:86% of the per-skipped-group cost.
The change
Load a group's bloom filters on first use inside the predicate loop rather than
up front. One call moves; nothing else changes. A group the zone map excludes now
reads none.
At 200,000 rows per group: per-skip 466.4 to 64.8, and the whole query from 9577
buffers to 1946, a factor of 4.9. The 64.8 matches the bloom-read-deleted
measurement, so no skipped group reads a bloom filter.
Results do not change. The filter was always a pruning step, never a source of an
answer.
Also tried, and it was not the cost
Adding
vector_index = -1as a scan key toColumnarReadZoneMapList, so theper-vector rows are filtered by the index instead of the loop. Per-skip went
466.4 to 427.9. Recorded here so it is not proposed again as a fix for this.
Test
test/bloom_lazy.sh, 14 checks, registered in the matrix.It measures block reads against
pgcolumnar.bloomwithpg_stat_get_blocks_fetched, not a total buffer count and not a wall clock, sothe thing it asserts is the thing that changed. The property: bloom reads follow
the groups a scan KEEPS, not the groups it examines. Two tables of 6 and 30
groups, both keeping exactly one group, must read about the same number of bloom
blocks.
A scan that keeps every group must read MORE, so a change that simply stopped
reading bloom filters cannot pass. Six parity checks against a heap mirror cover
equality on the segment key, equality no group holds, equality on a scattered
column, two equality predicates, a range predicate, and full set parity, plus two
with
pgcolumnar.enable_bloom_filter = off.Removal proof: restoring the eager read takes bloom blocks from 1 and 1 to 6 and
33 across the two tables, turning both property checks red.
What this does not settle
The 304,233 buffers reported in #310 are at 100M rows and 667 groups. My probes
are at 20 groups. Extrapolating this per-skip cost to that shape predicts most of
the number, but it is an extrapolation and I am not presenting it as that number
measured. #310 should stay open until the clustered q2 is re-run against this.
#314 covers the related waste this does not touch: even for a group the scan
keeps, bloom filters are read for every column rather than the columns carrying a
predicate. That is 464 of 715 buffers in the single-group baseline above.
Gates
Five-major matrix (15, 16, 17, 18.4, 19beta2).