perf(data-warehouse): probe target table for existence instead of information_schema

[EDsCODE]

Problem

The duckgres sink's _table_exists() runs SELECT 1 FROM information_schema.tables on every non-first batch (via _plan_batch_operation, to choose create vs insert/merge). On a large DuckLake catalog this forces duckgres to re-materialize its whole-catalog information_schema compat view — every current table and column at the current snapshot.

Traced end to end on the dogfood org's catalog (mw-prod-us, 2026-07): 3,173 tables, 80,235 snapshots, 577,076 ducklake_column rows (88% superseded version history), and the ducklake_column index is (table_id, …)-leading — so an "all current columns" load can't use it and scans the whole history.

• It's not a lock — sampling pg_stat_activity/pg_locks on the catalog Postgres during live sink activity showed zero blocking. It's work: DuckDB caches the catalog and re-loads it whenever the snapshot advances; the sink commits a snapshot per batch, so the cache is invalidated on every batch and each check re-scans the bloated catalog.
• Measured under identical concurrent write load: information_schema.tables = 1.3s idle → ~48s under load (44.8–52.1s every iteration, one concurrent writer was enough). This is a shared per-apply tax paid by every table, which is why sink throughput was flat (~140/hr) regardless of which table a batch hit — and why neither the earlier session-reuse nor co-claim PRs moved it (the cache is invalidated by other sessions' commits).

Change

Replace the information_schema enumeration with a single-table LIMIT 0 probe (SELECT 1 FROM <schema>.<table> LIMIT 0). It loads only that one table's metadata via the catalog's table_id-indexed single-table path, so it stays cheap even under concurrent snapshot commits:

under identical concurrent write load	idle	loaded
information_schema.tables (old)	1.3s	~48s
LIMIT 0 probe (new)	0.11s	0.59s (max 0.90s)

duckgres reports a missing table as a generic XX000 carrying DuckDB's stable "... does not exist" catalog message (verified via psycopg — it is not a typed UndefinedTable), so the probe matches that message for "absent" and propagates anything else — a transient error read as "table absent" would wrongly pick the create path (a benign, self-healing CREATE-then-retry, but avoid it). The probe runs on the autocommit connection outside any transaction, so a raised error is inert.

Kept the per-connection positive existence cache (WeakKeyDictionary): a table that exists stays existing for the connection's life (the sink never drops tables mid-run), so repeat batches skip the probe entirely and the not-found match only runs on a table's first encounter per connection.

How did you test this code?

I'm an agent; checks actually run:

• TDD: probe-succeeds→exists, probe-not-found→absent, other-error propagates (not read as absent), cache-hit skips the probe, per-table granularity, per-connection isolation, negatives-not-cached — each watched failing first, then passing (test_processor.py, 30 total).
• Direct production experiments established causation: idle vs concurrent-load repro (1.3s→48s for information_schema, 0.11s→0.59s for the probe), catalog Postgres pg_locks/index inspection, and the exact psycopg error signature. All test artifacts (a temporary user, a scratch schema) were removed.
• Pre-commit hooks (ruff/format/ty) clean.

Follow-ups (separate)

• Snapshot/version expiry to shrink the 577k-row catalog — makes even the rare create-path check cheap and helps storage/billing (this is the part of the earlier "compaction helps throughput" idea that's actually true — metadata expiry, not S3 file compaction).
• File compaction remains storage-only for throughput.

🤖 Agent context

Authored by Claude Code (Fable 5) from a multi-day production investigation. Notably this corrected an earlier hypothesis: the throughput bottleneck is not S3 file-count bloat (compaction) but this per-batch information_schema re-materialization — established by reproducing the 1.3s→48s swing under concurrent snapshot commits, confirming on the catalog Postgres it's re-read work (not lock contention), and then validating that the single-table probe stays sub-second under the same load.

🤖 Generated with Claude Code

[trunk-io]

😎 This pull request was merged.

[greptile-apps]

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/duckgres/processor.py:684-685
**Unrelated Missing Objects Look Absent**

The substring also matches errors such as a missing schema, catalog, or secret encountered while resolving the query. In those cases `_table_exists` returns `False`, so the batch selects the create path instead of propagating the real failure, producing a second misleading error and an unnecessary retry.

_{Reviews (1): Last reviewed commit: "perf(data-warehouse): probe target table..." | Re-trigger Greptile}

[greptile-apps]

Unrelated Missing Objects Look Absent

The substring also matches errors such as a missing schema, catalog, or secret encountered while resolving the query. In those cases _table_exists returns False, so the batch selects the create path instead of propagating the real failure, producing a second misleading error and an unnecessary retry.

Prompt To Fix With AI

This is a comment left during a code review.
Path: products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/duckgres/processor.py
Line: 684-685

Comment:
**Unrelated Missing Objects Look Absent**

The substring also matches errors such as a missing schema, catalog, or secret encountered while resolving the query. In those cases `_table_exists` returns `False`, so the batch selects the create path instead of propagating the real failure, producing a second misleading error and an unnecessary retry.

How can I resolve this? If you propose a fix, please make it concise.

[stamphog]

Note

🤖 stamphog reviewed 7531aca323699cfde30bc44b836f9d2864f72fae — verdict: REFUSED

Greptile's unresolved P1 comment is confirmed by the diff: the generic "does not exist" substring match will also swallow errors from a missing schema, catalog, or secret encountered while resolving the query, silently treating them as "table absent" and routing to the create path instead of surfacing the real failure — this is a substantive, unaddressed correctness concern in a data-ingestion path.

• Author wrote 100% of the modified lines and has 22 merged PRs in these paths (familiarity STRONG).
• greptile-apps[bot] reviewed the current head.
• Unresolved inline review comment from @greptile-apps on processor.py: the 'does not exist' substring match in _probe_table_exists is not specific to the target table and will also catch missing-schema/catalog/secret errors, causing the wrong (create) path to be chosen and masking a real failure.
• This is data-warehouse sync logic (create vs insert/merge path selection) — a misclassified error here is a data-integrity concern, not just cosmetic.

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	48L, 1F substantive, 111L/2F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1c-medium (111L, 2F, single-area, perf)
stamphog 2.0.0b3		.stamphog/policy.yml @ 54df693 · reviewed head 7531aca

[greptile-apps]

The concern is valid and worth addressing. The "does not exist" substring is indeed broad enough to catch errors like "Schema 'xyz' does not exist" or "Catalog 'abc' does not exist", routing those failures silently to the create path.

DuckDB's specific table-missing message (as verified by the test mock itself) is:

"Catalog Error: Table with name <name> does not exist!"

Since you already have duckgres_table in scope, you can anchor the match to that specific pattern and avoid the false-positive issue entirely:

    except psycopg.Error as err:
        if f"table with name {duckgres_table.lower()} does not exist" in str(err).lower():
            return False
        raise

This won't match schema/catalog/secret errors (their messages use different subjects — "Schema", "Catalog", etc.), and it stays robust to the wrapping prefix ("flight execute: ...") that duckgres prepends. The test mock already uses this exact format, so the existing tests cover it without changes.

[EDsCODE]

Addressed the P1 (@greptile-apps / @stamphog): the missing-table detection now anchors to DuckDB's specific "Table with name … does not exist" message rather than a bare "does not exist" substring, so a missing schema/catalog/secret encountered while resolving the query propagates as a real error instead of being read as "table absent" and routed to the create path. Added test_missing_schema_error_propagates_not_treated_as_table_absent (a Schema with name … does not exist error now raises). 8 existence tests pass.

[greptile-apps]

Good fix — the "table with name" in message and "does not exist" in message pattern is the right call here. A few observations:

1. The anchor is solid. DuckDB's catalog error format for a missing table is stable (Catalog Error: Table with name X does not exist), and the test directly exercises the distinction between Table with name vs Schema with name errors. The new test test_missing_schema_error_propagates_not_treated_as_table_absent closes the gap cleanly.

2. One subtle thing worth double-checking: the match is done on .lower(), but DuckDB's error message uses mixed case: "Table with name". Since you're matching "table with name" after .lower() this works — just confirming the lowercasing is intentional and consistent with how psycopg surfaces the message.

3. The test helper _existence_conn constructs the error string with the real DuckDB format "flight execute: ... Catalog Error: Table with name t does not exist!" — good, that confirms the match logic against the actual wire format rather than a made-up string.

Looks good overall. The fix is precise and the test coverage is thorough (8 cases exercising probe success, probe absent, cache hit, per-table granularity, per-connection isolation, negatives not cached, transient error propagation, and now missing schema propagation).

[stamphog]

Data-pipeline change to table-existence detection in warehouse sources; the one substantive review concern (a too-broad error-message match that could misroute a real failure to the create path) was fixed in a later commit with a specific new regression test, and Greptile's follow-up comment confirms the fix against the current code. Author has STRONG familiarity (100% of touched lines, 22 recent merged PRs in this path) which stands in for owning-team assurance despite not being formally on the team.

• Author wrote 100% of the modified lines and has 22 merged PRs in these paths (familiarity STRONG).
• 👍 on the PR from greptile-apps[bot], hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	51L, 1F substantive, 125L/2F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1c-medium (125L, 2F, single-area, perf)
stamphog 2.0.0b3		.stamphog/policy.yml @ 54df693 · reviewed head 9e527da

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-23 06:25 UTC	Run
prod-us	✅ Deployed	2026-07-23 06:44 UTC	Run
prod-eu	✅ Deployed	2026-07-23 06:46 UTC	Run