fix(plugin-backfill): add retry backoff and continue processing after chunk failure#80
Merged
fix(plugin-backfill): add retry backoff and continue processing after chunk failure#80
Conversation
… chunk failure Three fixes to backfill runtime: 1. Add exponential backoff between retries (configurable via defaults.retryDelayMs, default 1000ms) - Backoff formula: baseDelay * 2^(attempt-1), so retries happen at 1s, 2s, 4s, etc. - Allows transient ClickHouse Cloud merge conflicts to resolve before re-attempting 2. Don't stop backfill on first permanent chunk failure - continue processing remaining chunks - Failed chunks are marked failed and rest of plan continues - Final run status shows both done and failed chunk counts - Users can retry failed chunks with resume command 3. Resume always retries failed chunks - users shouldn't need --replay-failed - Simplifies the common recovery flow after transient errors Also refactor executeChunk to merge duplicated simulation/real execution branches. Docs: add retryDelayMs option to backfill docs, update resume behavior description, simplify failed chunk recovery example (no longer needs --replay-failed). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The backfill run now processes chunk 3 even when chunk 2 fails, so chunkCounts.done is 2 instead of 1. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
KeKs0r
force-pushed
the
marc/backfill-retry-backoff
branch
from
f4b3afc to
1daa9c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
defaults.retryDelayMs, default 1000ms) to recover from transient ClickHouse Cloud merge conflictsresumeautomatically retry failed chunks without requiring--replay-failedflagTest plan
🤖 Generated with Claude Code