Skip to content

fix(cleanup): reclaim superseded index files promptly - #8323

Merged
Xuanwo merged 1 commit into
mainfrom
gatekeeper/fix-8321-1
Aug 7, 2026
Merged

fix(cleanup): reclaim superseded index files promptly#8323
Xuanwo merged 1 commit into
mainfrom
gatekeeper/fix-8321-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • scan all managed index artifacts while manifest-derived UUID verification is still available
  • preserve referenced and recent-unverified index safety checks
  • add a multi-fragment regression test for short-retention cleanup

Root cause

Cleanup limited every managed subtree to files no newer than the earliest retained manifest. When that cutoff skipped a newer artifact under a superseded index UUID, the same cleanup pass could delete the old manifest that proved the UUID was committed. Later passes then treated the artifact as unverified and retained it for seven days.

Fix

Index cleanup now inspects the full managed index subtree. Referenced UUIDs remain protected, recent UUIDs with no manifest evidence remain protected, and superseded UUIDs verified by manifests scheduled for removal are reclaimed before that evidence disappears.

Validation

  • cargo test -p lance cleanup_recent_replaced_index_with_short_retention
  • cargo test -p lance dataset::cleanup::tests::cleanup_ (20 passed)
  • cargo fmt --all
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #8321

@github-actions github-actions Bot added the bug Something isn't working label Aug 6, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The index subtree is now classified while removable manifests can still verify superseded UUIDs, while referenced and recent unknown UUIDs retain their existing protections. Since timestamp filtering happens after recursive listing, this fixes the retention leak without adding object-store listing requests.

@Xuanwo
Xuanwo merged commit 1b504d9 into main Aug 7, 2026
40 checks passed
@Xuanwo
Xuanwo deleted the gatekeeper/fix-8321-1 branch August 7, 2026 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: a short cleanup_older_than strands index files behind the 7-day unverified threshold

1 participant