Skip to content

ingest/backfill.go: patch to always end backfill at checkpoint aligned ledger#610

Merged
cjonas9 merged 4 commits intomainfrom
end-at-checkpoint-aligned-ledger
Mar 16, 2026
Merged

ingest/backfill.go: patch to always end backfill at checkpoint aligned ledger#610
cjonas9 merged 4 commits intomainfrom
end-at-checkpoint-aligned-ledger

Conversation

@cjonas9
Copy link
Contributor

@cjonas9 cjonas9 commented Feb 22, 2026

What

Ensures the final right boundary of the fill/final filled ledger is always aligned to a checkpoint.

Why

Previously, despite checkpoint-aligning the right frontfilling bound in runFrontfill(), this bound would be re-set to the most recent ledger in the datastore in runFrontfillChunks(). This caused noisy logs as captive core would attempt to pull a future checkpoint that hadn't yet been published, that would be resolved when that checkpoint was published. Also, this caused a rare, nondeterministic bug, where a UNIQUE constraint violated error could occur under the following conditions:

  • backwards backfill was skipped (causing two frontfilling iterations).
  • in iteration 1 of frontfilling, the unaligned datastore tip advances from the first checkpoint into the next checkpoint.
  • the total amount of ledgers it advances by is < 64 ledgers.

This is unlikely in practice because, by the third bullet, it must be a small backfill (i.e. under one day of ledgers in the worst case).

Known limitations

N/A

@cjonas9 cjonas9 added the bug Something isn't working label Feb 22, 2026
@cjonas9 cjonas9 requested a review from a team February 23, 2026 14:46
@cjonas9 cjonas9 marked this pull request as ready for review February 23, 2026 15:11
Copilot AI review requested due to automatic review settings February 23, 2026 15:11
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR ensures the final frontfill/right boundary used during backfill is checkpoint-aligned, preventing frontfill from accidentally extending to an unaligned datastore tip and triggering noisy logs / rare uniqueness issues.

Changes:

  • Compute and persist bounds.frontfill.Last as the datastore tip aligned to the previous checkpoint in runFrontfill().
  • Stop frontfillChunks() from re-reading the latest datastore ledger and overriding the caller’s right bound.
  • Update log messages and advance frontfill.First based on the aligned right bound.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@cjonas9 cjonas9 merged commit f8c9a08 into main Mar 16, 2026
41 of 42 checks passed
@cjonas9 cjonas9 deleted the end-at-checkpoint-aligned-ledger branch March 16, 2026 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants