fix(data-warehouse): pin duckgres backfill promotions to their run generation - #69101
Merged
Conversation
…neration Both PRIMED promotions (the processor's post-swap mark_primed and the reconciler's chunks-complete CAS) filtered on state alone. A replan can retire run R1 and plan R2 while R1's final chunk is mid-swap — the duckgres transaction cannot be cancelled — so R1's late promotion flips R2's row to PRIMED, live batches unblock and apply onto the table R2's own swap later replaces, and those rows are silently dropped forever (their apply markers prevent any re-apply). Pin both CAS filters to backfill_run_uuid. Also guard the planner against in-flight replace-head live runs: planning mid-run pins a snapshot that covers the run's head (pre-applied), while the replace-head gate bypass lets its post-snapshot tail apply into the doomed pre-swap table — the swap then drops the tail with no failure signal. _plan_one now defers while such a run exists in the queue and retries on the next planner tick. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ll-generation-pinning
fuziontech
approved these changes
Jul 8, 2026
…ll-generation-pinning
EDsCODE
marked this pull request as ready for review
July 9, 2026 17:50
Contributor
|
Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/mas..." | Re-trigger Greptile |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two related holes in the duckgres backfill state machine, both found in a deep QA session on the consumer:
mark_primedand the reconciler's chunks-complete promotion CAS onstate=BACKFILLINGalone. A replan (replan_backfill, the documented prod remediation) can retire run R1 and plan R2 while R1's final chunk is still mid-swap - its duckgres transaction can't be cancelled. R1's late promotion then flips R2's row to PRIMED: live batches unblock and apply onto the pre-R2 table, R2's own swap later replaces that table, and the live rows are gone permanently (their apply markers prevent re-apply). Same exposure for a reconciler pod holding a stale state snapshot across another pod's replan._plan_onepins a Delta snapshot mid-run: the run's head batches get pre-applied as snapshot-covered, the replace-head gate bypass admits its post-snapshot tail into the doomed pre-swap table, and the swap discards those rows while their markers block re-apply. The schema ends PRIMED with data missing and no failure signal anywhere.Changes
mark_primedtakes the promoting run'srun_uuidand both promotion CAS filters addbackfill_run_uuid, so a stale generation's evidence can never promote a newer plan_plan_onedefers (returns, retried each planner tick) while a replace-head live run is in flight for the schema: batch 0 replace-shaped and delta-succeeded, with neither a duckgres-succeeded final marker nor a duckgres-failed batch for the runHow did you test this code?
I (Claude) added three tests to
test_backfill.py, each catching a regression nothing existing covered:mark_primedwith a stale run uuid must not flip a row that has moved to a new generation (and must flip with the matching one)backfill_run_uuidchanged after the pass's state snapshot was read_plan_onemust defer and stay unplanned while_has_inflight_replace_runis trueRan the duckgres backfill + processor suites: 27 passed (one existing processor test updated for the new
mark_primedsignature). The in-flight-run SQL mirrors the proven shape of_reconcile_needs_resync's completion-proof query against the same views.Automatic notifications
Docs update
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Deep QA session on the duckgres consumer (Claude Code): two independent review passes (a state-machine adversary and a lease/claim concurrency reviewer) converged on the replan-vs-slow-swap interleaving and the plan-during-replace data-loss path. The fix follows the module's own documented CAS discipline ("every transition is a compare-and-swap") - the promotions were the only transitions not pinned to the generation they had evidence for.