fix(data-warehouse): pause CDC schedule on non-retryable errors

[estefaniarabadan]

Problem

When a CDC source hits a non-retryable extraction error, we classify it correctly and Temporal stops retrying the activity. But the source's CDC extraction schedule keeps firing on its normal cadence. Each fire re-runs the same deterministic failure, and because a run that fails before its first flush backfills a terminal FAILED job per schema, this floods the Syncs tab with failed jobs for a source that cannot succeed until a human intervenes.

The loudest case is a bad-credentials (auth_failed) source: the schedule re-runs it constantly, generating a large share of all V3 CDC failures while the actual fix (correct the credentials) is entirely on the user's side.

Before this change, only slot_missing / slot_not_configured / publication_missing paused the schedule (via mark_cdc_broken). Every other non-retryable category kept firing.

Changes

In CDCExtractActivity._handle_failure, non-retryable errors that are not slot/publication problems now pause the source's CDC extraction schedule.

• Slot/publication errors keep going through mark_cdc_broken (they persist the cdc_broken marker and are eligible for Repair CDC, which recreates the slot).
• Everything else non-retryable (auth, SSL required, unreachable host, an unrecoverable decode/merge) calls a new _pause_cdc_extraction_schedule helper. It pauses the schedule best-effort and deliberately does not set cdc_broken: the slot is intact, so the source must stay ineligible for the destructive Repair CDC re-sync. The schema's FAILED status and the existing "then re-enable change data capture" message stay the recovery cue.

Note

Behavior change: a non-retryable CDC failure now pauses the source's extraction schedule. The user resumes by fixing the root cause and re-enabling CDC, which recreates an unpaused schedule (the same recovery the error message already points to). A genuinely transient failure would need a manual re-enable instead of self-healing on the next tick, but these categories are, by classification, not retryable, so that trade is worth stopping the churn.

Before:

flowchart TD
    F[Non-retryable CDC error] --> Q{Slot/publication missing?}
    Q -->|Yes| B[mark_cdc_broken: pause schedule + broken marker]
    Q -->|No| K[Fail schema only; schedule keeps re-firing]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    class B phBlue
    class K phRed

Loading

After:

flowchart TD
    F[Non-retryable CDC error] --> Q{Slot/publication missing?}
    Q -->|Yes| B[mark_cdc_broken: pause schedule + broken marker]
    Q -->|No| P[Pause extraction schedule; no broken marker]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    class B phBlue
    class P phYellow

Loading

How did you test this code?

Automated only. I am an agent and did no manual or production testing.

• Extended test_missing_slot_or_publication_marks_cdc_broken to assert the routing: an auth_failed failure (non-retryable, slot intact) pauses the schedule and does not call mark_cdc_broken, while slot/publication failures still mark broken and do not pause. This is the regression guard: revert the new branch and the auth case fails.
• Mocked the new _pause_cdc_extraction_schedule (it calls sync_connect) in the other non-retryable activity tests so they stay off the network.
• hogli test .../cdc/tests/test_extract_activity.py: 67 passed, 1 failure (TestBackpressureGuard::test_run_skips_tick_without_touching_schemas_or_reader). That failure is pre-existing: it reproduces on clean master, fails only in the full-file run with "Database access not allowed", and passes in isolation. It is an ordering/DB-fixture artifact unrelated to this change.
• hogli test .../cdc/tests/test_errors.py: 20 passed. ruff check and ruff format clean; hogli ci:preflight green.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

No docs change. This is internal pipeline behavior and the in-product recovery message is unchanged.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Directed by @estefaniarabadan; written by Claude (PostHog Code) in Claude Code. The problem surfaced during a 24h V3 pipeline health review that used the investigating-warehouse-sources-load skill; the fix used the /writing-tests gate before touching tests.

Decisions:

• Rejected reusing mark_cdc_broken for auth. cdc_broken is specifically the "slot/publication is gone, Repair CDC eligible" gate (repair.py:_require_broken_evidence recreates the slot and forces a full re-sync). Auth has a healthy slot, so a separate pause path was added that does not set the marker.
• Scoped to all non-retryable, non-slot/publication categories rather than auth alone, because they share the identical re-fire bug and the same "fix and re-enable" recovery. One existing test's own comment ("the run must stop rather than loop the schedule") already assumed this behavior that was not implemented.
• Rebased onto origin/master, which had just landed a slot_not_configured category on the same lines. My branch composes with it: that category is handled by mark_cdc_broken, and the new elif catches the rest.

---

Created with PostHog Code

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[greptile-apps]

_{Reviews (1): Last reviewed commit: "fix(data-warehouse): pause CDC schedule ..." | Re-trigger Greptile}

[github-actions]

🤖 CI report

⚠️ Backend coverage — 55.0% of changed backend lines covered — 8 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ███████████░░░░░░░░░ 55.0% (10 / 18)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/cdc/activities.py	27.3%	1194–1196, 1198, 1200–1203

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29502833558 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
warehouse_sources	███████████████████░ 96.3%	241,243 / 250,509

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-16 14:49 UTC	Run
prod-us	✅ Deployed	2026-07-16 15:12 UTC	Run
prod-eu	✅ Deployed	2026-07-16 15:12 UTC	Run