Skip to content

fix(data-warehouse): fail fast when CDC source has no replication slot - #71095

Merged
Gilbert09 merged 1 commit into
masterfrom
posthog-code/cdc-fail-fast-unconfigured-slot
Jul 15, 2026
Merged

fix(data-warehouse): fail fast when CDC source has no replication slot#71095
Gilbert09 merged 1 commit into
masterfrom
posthog-code/cdc-fail-fast-unconfigured-slot

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

Error tracking surfaced a Postgres CDC source failing with a misleading, dead-end error. The exception chain was:

  1. psycopg UndefinedObject: replication slot "" does not exist — note the empty slot name
  2. RuntimeError: Cannot recreate CDC replication slot: no slot name configured for this source
  3. NonRetryableException: The replication slot no longer exists on the source database ... Use Repair CDC to recreate it and re-sync.

The source had change data capture enabled but no replication slot name stored in its config. The extraction activity streamed with an empty slot name, which Postgres reports as replication slot "" does not exist. The slot-invalidation check reads that as a recoverable slot drop, so it runs the destructive recovery path (resets every CDC schema to snapshot, marks them failed) and then dead-ends in recreate_slot, which raises because there is no slot name to recreate.

The failure ends up classified as SLOT_MISSING, so the user is told to "Use Repair CDC". But Repair CDC also calls recreate_slot and fails the same way — there is no slot name. The guidance is a dead end.

Changes

Detect the missing-slot configuration up front and treat it as its own non-retryable category:

  • New CDCSlotNotConfiguredError and CDCErrorCategory.SLOT_NOT_CONFIGURED in the shared CDC error taxonomy, with actionable copy: disable and re-enable change data capture to set it up again.
  • The extraction activity guards for an empty slot name before streaming (_require_configured_slot). This skips the doomed empty-slot peek and the destructive slot-invalidation recovery entirely, and routes through the normal failure path so the source is still marked broken and the schedule stops firing.
  • SLOT_NOT_CONFIGURED is wired into the set of categories that mark the source broken, matching SLOT_MISSING / PUBLICATION_MISSING.

Behavior for a healthy source, and for a genuine slot-dropped source (which still correctly points at Repair CDC), is unchanged.

Note

This lives in the shared cdc/ layer, so it applies to both CDC engines (Postgres and Supabase). The guard is universally correct: a CDC source with no slot name can never stream.

How did you test this code?

Automated tests only (I, Claude, did not run this against a live source):

  • test_extract_activity.py::TestErrorClassification::test_missing_slot_name_fails_before_streaming_without_recovery — an empty-slot CDC source raises NonRetryableException with the new actionable message, never connects or streams, never calls recreate_slot, and marks the source broken with reason slot_not_configured. Catches a regression where the guard is removed and the empty-slot case again dead-ends at Repair CDC. Distinct from the existing missing-slot/publication test, which exercises a real slot name dropped on the source database (a different path).
  • test_errors.py::TestClassifyCDCError::test_slot_not_configured_is_classified_without_adapter and a new row in test_retryable_flag — lock in that the new exception classifies to SLOT_NOT_CONFIGURED and is non-retryable.

Ran locally: the two CDC test files plus TestSlotInvalidationRecovery and the Postgres CDC error tests all pass. ruff check and ruff format --check clean on the changed files.

🤖 Agent context

Autonomy: Fully autonomous

Authored by Claude Code (Opus 4.8) triaging a Postgres data-warehouse CDC error from PostHog error tracking. Invoked the /writing-tests skill before adding the regression tests.

Decision: this is a fixable bug, not a user/upstream error to add to NonRetryableErrors. The error was already non-retryable (it flows through the CDC classify_cdc_error path, not the batch get_non_retryable_errors path), so the remaining defect was the misleading dead-end guidance and the destructive recovery running on an unrecoverable config. Considered instead making recreate_slot regenerate a default slot name so Repair CDC could recover it, but rejected that as riskier: it would change recovery semantics and could orphan a real slot on a source that legitimately had a custom name that was lost.


Created with PostHog Code

A CDC-enabled source with an empty stored slot name streamed an empty slot
name, which Postgres reports as `replication slot "" does not exist`. The
invalidation check reads that as a recoverable slot drop, so the destructive
slot-invalidation recovery runs and then dead-ends because there is no slot
name to recreate. The failure surfaced as SLOT_MISSING, telling the user to
"Use Repair CDC" — but Repair CDC also needs a slot name and fails identically.

Guard the missing-slot configuration before streaming and classify it as a
distinct non-retryable category with guidance that actually resolves it
(disable and re-enable CDC). This skips the doomed peek and the dead-end
recovery while still pausing the schedule via the broken marker.

Generated-By: PostHog Code
Task-Id: 9c79eaf9-ebd2-41ab-a403-a7f647e2d3b1
@github-actions

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 15, 2026 14:25
@Gilbert09 Gilbert09 added the stamphog Request AI approval (no full review) label Jul 15, 2026 — with PostHog

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contained, well-tested fix to CDC error classification/fail-fast behavior in the warehouse-sources product; author is on the owning team, diff matches the description, and no data model/API/CI/dependency risk is introduced.

  • 👍 on the PR from hex-security-app[bot].
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list no deny categories matched
size 40L, 2F substantive, 112L/4F incl. docs/generated/snapshots — within ceiling
tier T1-agent / T1c-medium (112L, 4F, single-area, fix)
stamphog 2.0.0b3 .stamphog/policy.yml @ 40bff74 · reviewed head 62d6e37

@trunk-io

trunk-io Bot commented Jul 15, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@Gilbert09
Gilbert09 merged commit fbacf52 into master Jul 15, 2026
256 checks passed
@Gilbert09
Gilbert09 deleted the posthog-code/cdc-fail-fast-unconfigured-slot branch July 15, 2026 15:08
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 15, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-15 16:00 UTC Run
prod-us ✅ Deployed 2026-07-15 16:19 UTC Run
prod-eu ✅ Deployed 2026-07-15 16:21 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stamphog Request AI approval (no full review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants