fix(snowflake): stop the query timebomb cancelling long syncs

[Gilbert09]

Problem

Snowflake imports were failing with a ProgrammingError:

000604 (57014): <redacted>: SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled

Surfaced by error tracking: issue 019f5495. The raising frame is get_rows in snowflake.py, at streaming_cursor.execute(query, params) — the streaming data query for a table sync.

Root cause is our own config, not the customer's. We set network_timeout=300 on the connection to bound the per-request retry budget (so a stalled/half-open connection fails fast instead of hanging the worker thread). But the connector reuses network_timeout for a second, unrelated thing: a client-side query timebomb that arms a timer on cursor.execute() and cancels the running query once the timeout elapses (cursor.py: real_timeout = timeout or network_timeout, then cancel → ProgrammingError 000604/57014).

So the 300s network_timeout capped every query's total wall-clock at 300s. The import sync activity has a 6h start_to_close_timeout, and a large table's streaming SELECT ... ORDER BY (or the COUNT(*) estimate) can easily take longer than 5 minutes to execute — at which point the timebomb cancels it. Retrying can't help because it's a fixed function of table size, so this is a bug in our code, not a transient or upstream error.

Changes

Keep network_timeout for the retry budget, but stop it from cancelling long queries.

The retry budget is applied per HTTP request, so it never caps a query's total wall-clock — only the timebomb does. The connector lets you override the timebomb per query via cursor.execute(timeout=...). So the full-table data scans (the COUNT(*) estimate and the streaming SELECT) now pass an explicit, much larger per-query timeout, which overrides the timebomb for those queries while leaving the retry budget intact. That value mirrors the import activity's 6h start_to_close_timeout — Temporal's activity timeout is the real ceiling; the connector defers to it rather than pre-empting a sync that's still making progress.

-            cursor.execute(query, inner_query_args)
+            cursor.execute(query, inner_query_args, timeout=_SNOWFLAKE_QUERY_TIMEOUT_SECONDS)
...
-                    streaming_cursor.execute(query, params)
+                    streaming_cursor.execute(query, params, timeout=_SNOWFLAKE_QUERY_TIMEOUT_SECONDS)

Short metadata queries (schema/PK discovery) keep the default 300s timebomb — they're fast and run in the 10-minute discovery activity, so an early bound there is fine.

How did you test this code?

Automated tests only (I'm an agent — no manual Snowflake run):

• Extended TestBuildPipeline::test_builds_source_response_and_streams to assert the streaming scan runs with a per-query timeout above network_timeout — the regression guard: reusing the 300s network_timeout as the query cap is exactly what caused the incident.
• Extended TestGetRowsToSync::test_returns_count to assert the COUNT(*) probe carries the same long timeout (otherwise a large table trips the timebomb and the probe spuriously logs a 604).
• TestConnect::test_bounds_network_timeout still guards the retry-budget bound.
• Ran the full source test file: pytest .../snowflake/tests/test_snowflake.py — 97 passed.

🤖 Agent context

Autonomy: Fully autonomous

Triaged from an error-tracking webhook by Claude (PostHog Code). Confirmed the issue and frame via the error-tracking MCP tools, then read the installed snowflake-connector-python source to establish that network_timeout does double duty: per-request retry budget (network.py) and a per-query cancellation timebomb (cursor.py). Initially considered swapping to socket_timeout, but that drops the aggregate retry-budget protection (a flaky connection could then spin until the 6h Temporal activity timeout). The per-query execute(timeout=...) override keeps the retry budget and only lifts the cap where queries are legitimately long. Invoked the /writing-tests skill before touching tests.

---

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[greptile-apps]

_{Reviews (1): Last reviewed commit: "fix(snowflake): bound stalls with socket..." | Re-trigger Greptile}

[trunk-io]

     

Failed Test	Failure Summary	Logs
test_materialized_column_optimization_returns_correct_results_1_not_nullable	The test failed because two lists of items are not matching; one list has extra duplicate elements.	Logs ↗︎
test_does_not_throw_if_cannot_log_activity	The test encountered an IndexError because it tried to access an element in an empty list.	Logs ↗︎
test_materialized_column_optimization_returns_correct_results_0_nullable	The test failed because two lists of items are not matching; one list has extra duplicate elements.	Logs ↗︎
test_materialized_column_optimization_returns_correct_results_0_nullable	The test failed because two lists of items are not matching; one list has extra duplicate elements.	Logs ↗︎

... and 2 more

View Full Report ↗︎ ⋅ Docs

[stamphog]

Small, focused fix to a warehouse-sync timeout bug by an owning-team author with tests added; the one substantive reviewer concern (unbounded retry budget) was addressed in the current diff and acknowledged by Greptile's 👍, and this isn't schema/API/auth/billing/CI risky territory.

• Author wrote 67% of the modified lines and has 6 merged PRs in these paths (familiarity MODERATE).
• Gilbert09 reviewed the current head.

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	21L, 1F substantive, 30L/2F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1b-small (30L, 2F, single-area, fix)
stamphog 2.0.0b3		.stamphog/policy.yml @ 72f9c2a · reviewed head 5f97442

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-13 11:21 UTC	Run
prod-us	✅ Deployed	2026-07-13 11:34 UTC	Run
prod-eu	✅ Deployed	2026-07-13 11:36 UTC	Run