fix(data-warehouse): resolve duckgres backfill Delta path from url_pattern

[EDsCODE]

Problem

Newly-enabled incremental schemas from schema-qualified sources (Postgres public.foo) never prime into duckgres. Their backfill fails every retry with:

Generic delta kernel error: No files in log segment

delta_table_uri builds the Delta S3 path from schema.normalized_name, which keeps the source schema qualifier: public.posthog_hogfunction normalizes to public_posthog_hogfunction. But the v3 loader writes the Delta folder under the unqualified DLT resource name, posthog_hogfunction. So the backfill opens a prefix that has no _delta_log, delta-kernel reports "No files in log segment", and the schema retries forever until it hits the failure threshold and its live batches pile up in the failing-blocked bucket.

It only bites schema-qualified incremental sources: full_refresh tables never touch this path (they replace-write and prime immediately), and unqualified names (e.g. ad-report sources) already have normalized_name == folder. That's why it stayed latent since the backfill primer landed (#63144) and surfaced in prod-eu, where the internal project mirrors public.posthog_* app tables — 33 schemas stuck, all identical error.

Note

This is the source Delta read path only. The duckgres target table naming (e.g. postgres_eu_public_posthog_alertconfiguration) is a separate concern and is intentionally left unchanged.

Changes

delta_table_uri now takes the table's folder from the catalog url_pattern — the authoritative location the query engine already reads — instead of recomputing it from normalized_name. The environment-correct base (settings.BUCKET_URL + folder_path(), both shared with the writer) is kept as-is, so only the diverging leaf segment changes. Falls back to normalized_name when a schema has no catalog table yet (nothing to backfill).

-def delta_table_uri(schema: ExternalDataSchema) -> str:
-    return f"{settings.BUCKET_URL}/{schema.folder_path()}/{schema.normalized_name}"
+def delta_table_uri(schema: ExternalDataSchema) -> str:
+    return f"{settings.BUCKET_URL}/{schema.folder_path()}/{_delta_table_folder(schema)}"

I kept BUCKET_URL as the base rather than reconstructing the whole s3:// URI from url_pattern, because url_pattern is an https URL and parsing its host into a bucket differs across prod (virtual-hosted S3) and local dev (MinIO endpoint). Rebasing only the leaf fixes the exact divergence without host-parsing fragility.

How did you test this code?

I (Claude) ran these automated tests. No manual testing beyond the read-only prod-eu diagnostics that root-caused it (opening both the normalized_name path and the url_pattern path for the stuck schemas — the former failed, the latter opened with files).

Added test_backfill_snapshot.py::TestDeltaTableUri — an in-memory SimpleTestCase (no DB; exercises the real normalized_name and folder_path). What it catches that nothing did before: delta_table_uri had zero coverage, and this locks in that a schema-qualified source resolves the folder from url_pattern (posthog_hogfunction), not normalized_name (public_posthog_hogfunction) — the exact prod regression. Cases: schema-qualified, unqualified (agreement guard), trailing-glob url_pattern, and the no-table fallback.

Verified by mutation: reverting delta_table_uri to the old normalized_name form fails the schema_qualified and glob_suffix cases and leaves the other two green, so the test is pinned to the behavior, not the implementation.

• test_backfill_snapshot.py: 4 passed.
• test_backfill.py + test_processor.py + test_backfill_snapshot.py: 48 passed (no regression from the changed URI).

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

Not applicable. Internal sink-path fix, no user-facing or documented behavior change.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Claude (Claude Code, Opus 4.8) root-caused this from a prod-eu dashboard showing 499 failing-blocked batches. Read-only diagnostics on the warehouse-sources-duckgres-load pod narrowed it: all 33 stuck schemas shared one error, region and bucket were correct (ruling out a region gap), and opening the Delta table at the catalog url_pattern succeeded while the sink's computed path failed. That isolated the divergence to delta_table_uri using normalized_name.

Decisions: fix the source read path only, not the duckgres target naming (separate, working, intentional). Derive the leaf from url_pattern rather than reconstructing the full s3:// URI from its https host (fragile across prod S3 vs local MinIO). Standalone PR rather than folding into #69098 ("unify duckgres data-import table naming"), which is already approved and concerns the target-write path — this is the source-read path in a different file.

Skills invoked: /writing-tests (to gate and shape the regression test).

[greptile-apps]

_{Reviews (1): Last reviewed commit: "fix(data-warehouse): resolve duckgres ba..." | Re-trigger Greptile}

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-11 00:34 UTC	Run
prod-us	✅ Deployed	2026-07-11 00:45 UTC	Run
prod-eu	✅ Deployed	2026-07-11 00:46 UTC	Run