fix(data-warehouse): stop stripe syncs failing on mispersisted incremental fields

[Gilbert09]

Problem

The warehouse job-failure dashboard shows a steady stream of Stripe syncs failing with KeyError: 'Field "created" does not exist in schema' (and a created_at variant) — hundreds of failed jobs across dozens of schemas in the last week, each retrying on every scheduled sync. The KeyError is pyarrow's, raised at incremental-cursor extraction when the configured incremental_field isn't a column in the extracted rows.

Two ways the bad config got there:

1. Display label persisted instead of the field. Schema discovery surfaces Stripe's incremental fields as label: "created_at" / field: "created". The source-creation endpoint persisted whatever string the caller sent, so API callers who echoed the label back got incremental_field: "created_at" (with the label's datetime type instead of the field's integer) — and every sync then crashed.
2. Migration 0794 set the wrong field on InvoiceItem. It blanket-set incremental_field: "created" on all Stripe append schemas missing one, but InvoiceItem's only incremental field is date — Stripe invoice items don't return a created column.

Changes

• Canonicalize at creation (external_data_source.py): when the source's discovery declares incremental fields for a schema, the payload's incremental_field is matched against both field and label, and the declared field + real field_type are persisted. Unknown values pass through unchanged (no behavior change for SQL sources, whose eligible-column lists aren't exhaustive).
• Actionable, non-retryable runtime error (load.py): get_incremental_field_value now raises IncrementalFieldMissingFromDataError — naming the missing field and listing available columns — instead of a raw pyarrow KeyError. The message is registered in Any_Source_Errors, so any source hitting it pauses the schema with user guidance ("pick a valid incremental field") rather than replaying the same failure on every scheduled sync forever.
• Data migration (0070) healing existing Stripe schemas: created_at → created/integer on every resource (matching what Stripe declares), and InvoiceItem created/created_at → date/integer. A stashed incremental_field_last_value that can't serve as an epoch cursor under the corrected integer type (e.g. an ISO datetime string) is cleared so the next sync starts a clean pass; numeric cursors are kept.

How did you test this code?

Automated tests I (Claude) ran locally, all green:

• test_load.py — two new tests: the missing-column case raises the actionable error (not a raw KeyError) and its message stays matched by an Any_Source_Errors entry — this cross-link is load-bearing (message drift would silently break the pause-with-guidance behavior and revert to infinite retries), and no other test covers it. Plus a happy-path max-cursor case for the changed lookup. Full file: 10 passed.
• test_external_data_source.py — new test_create_canonicalizes_incremental_field_label_to_declared_field: posting a Stripe schema with incremental_field: "created_at" / "datetime" (exactly what broken callers send) persists created/integer. Catches the canonicalization being removed, which no existing test covers.
• manage.py makemigrations --check --dry-run: no state drift. The migration mirrors the repo's data-migration pattern (RunPython + noop reverse, elidable) and touches thousands of rows at most, no locks.

Verified against production data (read-only): the failing schemas' sync_type_config shows exactly the two flavors above; the migration's filters match them and nothing else.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

No user-facing docs change — this fixes config handling and error reporting; the sync-method UI flow is unchanged.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Claude Code session, directed by Tom, starting from the job-failure dashboard tile. Investigation path: dashboard insight → job mirror tables for the full latest_error and affected schema configs → matched the persisted values against the Stripe source's declared incremental fields (label vs field) and traced flavor 2 to migration 0794's blanket backfill. Skills invoked: /django-migrations, /writing-tests, /debugging-ci-failures conventions for the read-only investigation. Considered also validating incremental_field on the schema-update PATCH path, but that path is driven by a dropdown fed from the declared-fields endpoint (so it sends the right value) and validating there would require re-running source discovery per PATCH; left as a possible follow-up.

[github-actions]

🤖 CI report

⚠️ Backend coverage — 82.0% of changed backend lines covered — 11 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████░░░░ 82.0% (52 / 63)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/migrations/0070_fix_stripe_incremental_fields.py	47.6%	16–21, 35–39

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29412686412 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
tasks	█████████████░░░░░░░ 67.4%	25,449 / 37,779
signals	████████████████░░░░ 79.1%	19,072 / 24,106
data_modeling	████████████████░░░░ 80.0%	4,834 / 6,045
cdp	████████████████░░░░ 80.7%	3,118 / 3,864
agent_platform	█████████████████░░░ 84.2%	3,112 / 3,695
notebooks	█████████████████░░░ 84.3%	6,323 / 7,501
cohorts	█████████████████░░░ 86.0%	3,989 / 4,639
actions	█████████████████░░░ 86.6%	717 / 828
product_tours	█████████████████░░░ 87.5%	1,266 / 1,447
exports	██████████████████░░ 88.3%	6,861 / 7,769
conversations	██████████████████░░ 88.9%	16,126 / 18,130
dashboards	██████████████████░░ 89.0%	5,647 / 6,345
mcp_analytics	██████████████████░░ 89.1%	2,502 / 2,807
error_tracking	██████████████████░░ 89.5%	9,683 / 10,816
engineering_analytics	██████████████████░░ 89.8%	4,861 / 5,414
streamlit_apps	██████████████████░░ 90.4%	2,499 / 2,764
slack_app	██████████████████░░ 90.6%	9,511 / 10,503
marketing_analytics	██████████████████░░ 90.8%	11,514 / 12,684
alerts	██████████████████░░ 90.9%	3,416 / 3,760
product_analytics	██████████████████░░ 91.1%	5,527 / 6,068
workflows	██████████████████░░ 92.0%	4,795 / 5,210
data_warehouse	██████████████████░░ 92.1%	17,885 / 19,428
web_analytics	███████████████████░ 92.7%	13,607 / 14,674
ai_observability	███████████████████░ 92.8%	14,771 / 15,923
surveys	███████████████████░ 92.9%	5,660 / 6,094
posthog_ai	███████████████████░ 93.2%	1,312 / 1,408
approvals	███████████████████░ 93.3%	3,395 / 3,640
reminders	███████████████████░ 93.4%	468 / 501
early_access_features	███████████████████░ 93.8%	848 / 904
endpoints	███████████████████░ 94.1%	8,606 / 9,143
skills	███████████████████░ 94.4%	2,827 / 2,995
revenue_analytics	███████████████████░ 94.5%	3,598 / 3,809
review_hog	███████████████████░ 94.6%	6,475 / 6,848
logs	███████████████████░ 95.3%	9,525 / 9,991
experiments	███████████████████░ 95.6%	24,138 / 25,244
replay_vision	███████████████████░ 95.7%	13,300 / 13,896
feature_flags	███████████████████░ 96.1%	14,750 / 15,354
warehouse_sources	███████████████████░ 96.2%	219,841 / 228,591
annotations	███████████████████░ 96.2%	732 / 761
user_interviews	███████████████████░ 96.4%	2,242 / 2,325
data_catalog	███████████████████░ 97.1%	2,034 / 2,095
customer_analytics	███████████████████░ 97.3%	7,442 / 7,648

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

⚠️ Django migration SQL — 1 new migration to review

We've detected new migrations on this PR. Review the SQL output for each migration:

products/warehouse_sources/backend/migrations/0070_fix_stripe_incremental_fields.py

BEGIN;
--
-- Raw Python operation
--
-- THIS OPERATION CANNOT BE WRITTEN AS SQL
COMMIT;

Last updated: 2026-07-15 11:47 UTC (67ee026)

✅ Django migration risk — migration analysis complete

We've analyzed your migrations for potential risks.

Summary: 0 Safe | 1 Needs Review | 0 Blocked

⚠️ Needs Review

May have performance impact

warehouse_sources.0070_fix_stripe_incremental_fields
  └─ #1 ⚠️ RunPython: RunPython data migration needs review for performance

📚 How to Deploy These Changes Safely

RunPython:

Use batching for large data migrations:

• Use .iterator() to avoid loading all rows into memory
• Use .bulk_update() instead of saving individual objects
• Batch size: 1,000-10,000 rows per batch
• Add pauses between batches
• Consider background jobs for very large updates (millions of rows)

See the migration safety guide

Last updated: 2026-07-15 11:47 UTC (67ee026)

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-15 13:34 UTC	Run
prod-us	✅ Deployed	2026-07-15 14:01 UTC	Run
prod-eu	✅ Deployed	2026-07-15 14:01 UTC	Run