feat(data-warehouse): split failing backfills out of the duckgres blocked backlog - #68785
Conversation
…cked backlog
A schema whose backfill fails persistently (planner error or a terminally
failed backfill run) previously kept its blocked batches on the pageable
blocked-backlog gauges forever, pinning the oldest-age panel and hiding
genuine throughput problems - and once its batches aged past queue
retention the failure disappeared entirely. The planner also retried the
broken schema every ~30s with no backoff.
- DuckgresSinkSchemaState gains consecutive_failures + first_failed_at;
every failed attempt records the streak, any forward progress resets it
- failing := streak >= 3 (or parked in needs_resync); capacity reverts and
supersession stay neutral
- planner retries failing schemas with capped exponential backoff
(30s -> 30min, never gives up)
- blocked gauges now count healthy in-progress schemas only (pageable);
failing schemas' batches move to duckgres_sink_failing_blocked_backlog
(unalerted)
- new state-derived duckgres_sink_stuck_backfill{,_oldest_age_seconds}
gauges survive queue retention - the durable backfill-owed signal
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migration SQL ChangesHey 👋, we've detected some migrations on this PR. Here's the SQL output for each migration, make sure they make sense:
|
🔍 Migration Risk AnalysisWe've analyzed your migrations for potential risks. Summary: 1 Safe | 0 Needs Review | 0 Blocked ✅ SafeBrief or no lock, backwards compatible 📚 How to Deploy These Changes SafelyAddField: This operation acquires a brief lock but doesn't rewrite the table. Deployment uses lock timeouts with automatic retries, so lock contention will cause retries rather than connection pile-up. Last updated: 2026-07-09 20:03 UTC (befaf94) |
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| def _revert_to_pending(state_id: Any, error: str | None = None, *, record_failure: bool = False) -> None: | ||
| """record_failure distinguishes a failed attempt from a neutral revert | ||
| (capacity re-check lost the race) — capacity is pacing, not failure, and | ||
| must not start a streak.""" | ||
| now = timezone.now() | ||
| updates: dict[str, Any] = { | ||
| "state": DuckgresSinkSchemaState.State.PENDING_BACKFILL, | ||
| "updated_at": timezone.now(), | ||
| "updated_at": now, | ||
| } | ||
| if error is not None: | ||
| updates["last_error"] = error | ||
| if record_failure: | ||
| updates.update(_failure_streak_updates(now=now)) | ||
| DuckgresSinkSchemaState.objects.filter( | ||
| id=state_id, state=DuckgresSinkSchemaState.State.BACKFILLING, backfill_run_uuid__isnull=True | ||
| ).update(**updates) |
There was a problem hiding this comment.
_revert_to_pending always sets updated_at = now, even on the neutral capacity path (record_failure=False). The backoff window in _plan_pending is computed as timezone.now() < state.updated_at + backoff_seconds, so a schema that just passed its 30-minute backoff window, got capacity-reverted, and went back to PENDING will see its anchor reset to NOW — meaning it must wait another full backoff window before the planner will attempt it again.
In a consistently capacity-constrained environment (pre-check passes, post-claim check fails repeatedly) a schema near the cap could have its 30-minute window restarted on every planner tick, preventing it from ever retrying despite the failure streak counting up. The fix is to skip the updated_at write in the neutral-revert path when the schema already has an active streak (consecutive_failures > 0), or to track the backoff anchor separately from updated_at.
…failing-backfill-tracking
…failing-backfill-tracking
…k tests Satisfies the test-datetime-now-without-freeze semgrep rule, which landed on master after this branch was cut. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…and tests updated_at is nullable to the type checker (auto_now + null=True), so guard it before the backoff arithmetic. Route the fake queue connection through a typed factory instead of passing it straight to _reconcile_one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Reviews (2): Last reviewed commit: "fix(data-warehouse): satisfy mypy in duc..." | Re-trigger Greptile |
🤖 CI report |
|
⏭️ Skipped snapshot commit because branch advanced to The new commit will trigger its own snapshot update workflow. If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:
|
2 participants
Problem
The duckgres sink's blocked-backlog gauges conflate two very different things: batches waiting behind a schema whose backfill is progressing normally (drains on its own), and batches waiting behind a schema whose backfill fails persistently (never drains without intervention). One wedged schema pins
duckgres_sink_blocked_oldest_age_secondsfor days, so the gauge can neither carry a page nor distinguish a real throughput problem from a single broken table.Two aggravating factors:
Changes
Failure lifecycle on
DuckgresSinkSchemaState(migration 1248 addsconsecutive_failures,first_failed_at):first_failed_atonce per streak. Any forward progress (successful plan, chunk applied, primed) resets both pluslast_error.needs_resync. Capacity reverts, supersession by a replace run, and operatorreplan_backfillare streak-neutral.failedstatuses are terminal, so there's no retry loop to wait out, and without this a mid-backfill wedge would sit in the pageable bucket forever.Retry backoff: the planner skips a failing schema still inside
min(30s * 2^(failures-1), 30min)(jittered) since its last attempt. It never gives up - at the cap that's one attempt per ~30min, so a schema self-heals the moment the underlying issue clears.Metric split (batch-derived, retention-bounded):
duckgres_sink_blocked_backlog/_oldest_age_secondsduckgres_sink_failing_blocked_backlog(new)Durable stuck gauges (state-derived, survive queue retention):
duckgres_sink_stuck_backfill(count of failing schemas) andduckgres_sink_stuck_oldest_age_seconds(age of the oldestfirst_failed_at). Deliberately unalerted: they're an operator remediation queue and the data source for a future UI surface, not a page.How did you test this code?
Automated tests only (I'm an agent; no manual prod testing). Full duckgres
pipeline_v3suite passes locally (99 tests). New tests, each catching a regression nothing covered before:TestFailureStreak- streak records once per attempt with a stablefirst_failed_atanchor;BackfillUnsupportedErrorparks with a streak; capacity reverts stay neutral (a busy org must not misclassify); backoff skips inside the window and retries after it;mark_primed/replan_backfillreset the streak; reconcile escalates a terminally failed run to failing exactly once (the silent-suppression gap), stays neutral on supersession, and resets on chunk progress.TestFailingSchemaClassification- the failing predicate over state x streak combinations, including thatprimednever classifies.test_backlog_stats- failing schemas' batches leave the pageable blocked bucket (count and oldest-age) and land in the failing bucket; a failing-but-eligible schema is not counted (failing splits the blocked set only).Migration applies cleanly;
makemigrations --checkshows no drift.Automatic notifications
Docs update
Internal consumer/metrics behavior; no user-facing docs affected.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
/django-migrations(additive columns, no external writers of the table so nodb_defaultneeded) and/writing-tests(value gate for the new test classes).transaction=Trueclass becausefailing_schema_idsdoes thread-entryclose_old_connections(), which severs the transaction-wrapped test connection.🤖 Generated with Claude Code