ingest/backfill.go: patch to always end backfill at checkpoint aligned ledger#610
Merged
ingest/backfill.go: patch to always end backfill at checkpoint aligned ledger#610
Conversation
There was a problem hiding this comment.
Pull request overview
This PR ensures the final frontfill/right boundary used during backfill is checkpoint-aligned, preventing frontfill from accidentally extending to an unaligned datastore tip and triggering noisy logs / rare uniqueness issues.
Changes:
- Compute and persist
bounds.frontfill.Lastas the datastore tip aligned to the previous checkpoint inrunFrontfill(). - Stop
frontfillChunks()from re-reading the latest datastore ledger and overriding the caller’s right bound. - Update log messages and advance
frontfill.Firstbased on the aligned right bound.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Shaptic
approved these changes
Mar 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ensures the final right boundary of the fill/final filled ledger is always aligned to a checkpoint.
Why
Previously, despite checkpoint-aligning the right frontfilling bound in runFrontfill(), this bound would be re-set to the most recent ledger in the datastore in runFrontfillChunks(). This caused noisy logs as captive core would attempt to pull a future checkpoint that hadn't yet been published, that would be resolved when that checkpoint was published. Also, this caused a rare, nondeterministic bug, where a
UNIQUE constraint violatederror could occur under the following conditions:This is unlikely in practice because, by the third bullet, it must be a small backfill (i.e. under one day of ledgers in the worst case).
Known limitations
N/A