Clean up stale star-tree index leftover from killed prior build - #19331
Merged
Jackie-Jiang merged 1 commit intoAug 23, 2026
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #19331 +/- ##
============================================
+ Coverage 66.97% 67.06% +0.08%
Complexity 1424 1424
============================================
Files 3463 3467 +4
Lines 222078 222329 +251
Branches 34957 34996 +39
============================================
+ Hits 148743 149103 +360
+ Misses 61467 61318 -149
- Partials 11868 11908 +40
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Jackie-Jiang
added
bug
Jackie-Jiang
approved these changes
Aug 21, 2026
Jackie-Jiang
left a comment
Jackie-Jiang
left a comment
Contributor
There was a problem hiding this comment.
Just minor comments
deepthi912
force-pushed
the
fix/startree-stale-index-cleanup
branch
from
August 22, 2026 21:30
b74de8d to
b90d348
Compare
MultipleTreesBuilder.build() opens StarTreeIndexCombiner which requires the target star_tree_index file to NOT exist (StarTreeIndexCombiner:47). A previous build that was hard-killed (JVM crash, container OOM, thread hard-interrupt) can leave a partial star_tree_index at the segment root without a matching STAR_TREE_COUNT in segment metadata, because the metadata save at line 265 runs only after all trees complete. The catch block at line 249-262 cleans up on Java exceptions but never runs on process kill. On the next preprocess, segment metadata says "no star-tree" so `_separator == null`, the incremental move-aside step is skipped, and the combiner opens on the leftover file, hitting the checkState. The failure is not self-healing: the leftover keeps blocking every retry. Fix: when `_separator == null` (fresh build, no matching metadata), delete any stale star_tree_index / star_tree_index_map / EXISTING_STAR_TREE_TEMP_DIR before opening the combiner. Metadata is authoritative — with `_separator == null`, any on-disk star-tree artifact is orphaned. Log at WARN so operators see the recovery. The incremental path (`_separator != null`) is untouched. Adds MultipleTreesBuilderStaleCleanupTest covering: - stale star_tree_index at segment root -> cleaned up + fresh tree built - stale EXISTING_STAR_TREE_TEMP_DIR -> cleaned up + fresh tree built
deepthi912
force-pushed
the
fix/startree-stale-index-cleanup
branch
from
August 22, 2026 21:40
b90d348 to
7320a7a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MultipleTreesBuilder.build()opensStarTreeIndexCombiner, which requiresstar_tree_indexto not already exist (StarTreeIndexCombiner.java:47).If a previous build is hard-killed (JVM crash, container OOM, thread interrupt) after the combiner opens but before the metadata save at line 265,
star_tree_indexis left on disk while segment metadata has noSTAR_TREE_COUNT. Every subsequent preprocess on that segment throwsIllegalStateException: Star-tree index file already exists— not self-healing.Fix
Before opening the combiner, if
_separator == null(no matching star-tree in metadata → any on-disk artifact is orphaned), deletestar_tree_index,star_tree_index_map, and any lingeringEXISTING_STAR_TREE_TEMP_DIR. The incremental path (_separator != null) is untouched.Tests
MultipleTreesBuilderStaleCleanupTest— both fail without the fix.