Skip to content

fix(core): compute bitmap build gap on its own index type, not btree#480

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JunRuiLee:fix/btree-index-type-const-merge-residual
Jul 8, 2026
Merged

fix(core): compute bitmap build gap on its own index type, not btree#480
JingsongLi merged 1 commit into
apache:mainfrom
JunRuiLee:fix/btree-index-type-const-merge-residual

Conversation

@JunRuiLee

@JunRuiLee JunRuiLee commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Purpose

Follow-up to #481. #481 fixed the compile break on main (the undefined BTREE_INDEX_TYPE from the #478/#479 merge) by renaming the four references to BTREE_GLOBAL_INDEX_TYPE. This PR fixes a second, subtler bug on the same two call sites that the rename alone left latent.

Since #478, BTreeGlobalIndexBuildBuilder builds btree or bitmap indexes (resolved via normalize_sorted_global_index_type(self.index_type)). But the incremental gap computation (indexed_row_ranges) and the overlap guard (validate_existing_index_overlap) in execute() pass the hard-coded btree constant instead of the builder's resolved type. So a bitmap build computes its gap over btree coverage: if a btree index already covers the same field, the bitmap build treats those rows as already indexed and skips them, producing an incomplete or empty bitmap index. No test caught it because the only bitmap test covered a first full build.

Brief change log

  • Point the two execute() call sites (indexed_row_ranges gap at ~line 116 and validate_existing_index_overlap guard at ~139) at the builder's resolved index_type variable, so btree builds compute over btree coverage and bitmap builds over bitmap coverage. This makes bitmap builds correctly incremental through the shared builder — matching Java, where btree and bitmap share the sorted incremental path.
  • Add bitmap tests: second-build-is-noop, incremental-only-new-rows (append-only; new file range in the gap), and btree-coexistence (a bitmap build over a field already covered by a btree index still indexes all rows). The coexistence test is the regression guard — it fails if the type is hard-coded.

Tests

  • cargo test -p paimon --lib (3 new bitmap tests) passes; cargo build -p paimon clean.
  • Red check: reverting the fix (hard-coding btree) makes bitmap_build_after_btree_on_same_field_still_indexes fail, confirming the test catches the bug.
  • cargo fmt --all --check and cargo clippy -p paimon --all-targets -- -D warnings clean.

API and Format

No API or format change. IndexFileMeta.index_type on written index files was already dynamic (#478) and is unaffected; this only corrects which coverage the incremental gap/guard reads.

Documentation

None.

@JunRuiLee JunRuiLee changed the title fix(core): repair main build broken by #478/#479 merge residual (BTREE_INDEX_TYPE) fix(core): repair main build + bitmap incremental gap (#478/#479 merge) Jul 8, 2026
The incremental gap computation and overlap guard in the sorted global-index
builder's execute() hard-coded the btree index type. Since apache#478 the same builder
also builds bitmap indexes (via normalize_sorted_global_index_type), so a bitmap
build was computing its gap over BTREE coverage: if a btree index already covered
the same field, the bitmap build would treat those rows as already indexed and
skip them, producing an incomplete or empty bitmap index. Pass the builder's
resolved index_type instead, so bitmap builds are incremental over bitmap
coverage. Adds bitmap no-op / incremental / btree-coexistence tests (the
coexistence test fails if the type is hard-coded).
@JunRuiLee JunRuiLee force-pushed the fix/btree-index-type-const-merge-residual branch from 45cf628 to e44a751 Compare July 8, 2026 10:04
@JunRuiLee JunRuiLee changed the title fix(core): repair main build + bitmap incremental gap (#478/#479 merge) fix(core): compute bitmap build gap on its own index type, not btree Jul 8, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The fix correctly uses the resolved sorted global index type for both incremental coverage computation and overlap validation, so bitmap builds are no longer keyed against btree coverage.

I also checked the added regression coverage for bitmap no-op rebuilds, bitmap-after-btree coexistence on the same field, and incremental bitmap builds. Verified locally with cargo test -p paimon --lib bitmap_; CI is green.

@JingsongLi JingsongLi merged commit eaf4e51 into apache:main Jul 8, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants