fix(clickhouse): wait for mutation capacity before enqueueing and cap…

[bciaraldi]

… retry backoff## Problem

Weekly person overrides squash (and the deletes run it triggers on success) fails when a long-running mutation like a backfill is already on the target table. ClickHouse rejects the new mutation with TOO_MANY_MUTATIONS and the run dies. Separately, retry backoff between attempts grows uncapped.

Changes

• MutationRunner now polls system.mutations every 60s and waits for the target table to have no unfinished mutations before enqueueing. Jobs run late instead of failing.
• MutationRunner retries the enqueue through the capacity wait when ClickHouse rejects it with TOO_MANY_MUTATIONS (692), covering the race where another mutation lands between the capacity check and the ALTER. Handled in the runner itself so Temporal callers without a retry policy are covered too.
• Capped ExponentialBackoff retry delays at 60s across all Dagster ClickHouse cluster resources.
• Made retry_max_attempts configurable on ClickhouseClusterResource (default 8), raised to 20 for scheduled squash and sensor-triggered deletes runs.

How did you test this code?

Reproduced both failure modes from scheduled run history: immediate TOO_MANY_MUTATIONS on enqueue while a backfill mutation was running, and retry exhaustion from socket timeouts during shard waits. Existing MutationRunner tests in posthog/clickhouse/test/test_cluster.py cover the enqueue path (the new wait returns immediately when the table has no unfinished mutations). No manual testing against a live cluster.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

Not needed, internal operational behavior only.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Claude (Cowork session) diagnosed the failures from Dagster run history and authored the changes under Bryan's direction. Considered a Dagster op-level retry policy and a sensor-level pre-check, rejected both: op retries are bounded and don't fit hours-long backfills, and a launch-time check misses mutations that start mid-run. The wait lives in MutationRunner so squash, deletes, and sessions cleanup all benefit.

[greptile-apps]

_{Reviews (1): Last reviewed commit: "Merge branch 'master' of github.com:Post..." | Re-trigger Greptile}

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[github-actions]

🤖 CI report

✅ Playwright — all passed

All tests passed.

View test results →

⚠️ Backend coverage — 97.0% of changed backend lines covered — 1 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ███████████████████░ 97.0% (39 / 40)

File	Patch	Uncovered changed lines
posthog/clickhouse/cluster.py	94.4%	780

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29273239175 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
platform_features	██░░░░░░░░░░░░░░░░░░ 12.1%	7 / 58
batch_exports	████████░░░░░░░░░░░░ 39.6%	8,347 / 21,104
demo	███████████░░░░░░░░░ 55.2%	1,436 / 2,601
warehouse_sources_queue	████████████░░░░░░░░ 59.2%	148 / 250
tasks	█████████████░░░░░░░ 66.7%	24,376 / 36,570
data_tools	██████████████░░░░░░ 70.0%	63 / 90
ai_gateway	███████████████░░░░░ 75.0%	9 / 12
data_modeling	████████████████░░░░ 78.4%	4,696 / 5,987
signals	████████████████░░░░ 78.5%	18,070 / 23,028
cdp	████████████████░░░░ 80.6%	3,105 / 3,851
wizard	████████████████░░░░ 82.5%	772 / 936
cohorts	█████████████████░░░ 82.9%	3,048 / 3,675
notebooks	█████████████████░░░ 83.8%	6,086 / 7,259
agent_platform	█████████████████░░░ 84.1%	3,095 / 3,678
actions	█████████████████░░░ 86.6%	717 / 828
engineering_analytics	█████████████████░░░ 87.1%	4,180 / 4,798
product_tours	█████████████████░░░ 87.5%	1,266 / 1,447
exports	██████████████████░░ 88.3%	6,857 / 7,763
visual_review	██████████████████░░ 88.5%	5,565 / 6,287
business_knowledge	██████████████████░░ 88.5%	4,400 / 4,969
conversations	██████████████████░░ 88.9%	15,924 / 17,921
mcp_analytics	██████████████████░░ 89.1%	2,485 / 2,790
dashboards	██████████████████░░ 89.1%	5,650 / 6,342
error_tracking	██████████████████░░ 89.2%	9,473 / 10,615
streamlit_apps	██████████████████░░ 90.4%	2,499 / 2,764
slack_app	██████████████████░░ 90.6%	9,460 / 10,444
links	██████████████████░░ 90.6%	183 / 202
marketing_analytics	██████████████████░░ 90.7%	11,476 / 12,646
product_analytics	██████████████████░░ 91.2%	5,621 / 6,164
managed_migrations	██████████████████░░ 91.9%	908 / 988
workflows	██████████████████░░ 92.0%	4,795 / 5,210
mcp_store	██████████████████░░ 92.1%	3,661 / 3,977
alerts	██████████████████░░ 92.1%	3,389 / 3,678
data_warehouse	██████████████████░░ 92.2%	17,217 / 18,670
web_analytics	███████████████████░ 92.7%	13,702 / 14,787
notifications	███████████████████░ 92.7%	1,026 / 1,107
ai_observability	███████████████████░ 92.7%	14,670 / 15,822
surveys	███████████████████░ 92.9%	5,660 / 6,094
posthog_ai	███████████████████░ 93.2%	1,311 / 1,407
tracing	███████████████████░ 93.2%	2,423 / 2,599
approvals	███████████████████░ 93.3%	3,395 / 3,640
reminders	███████████████████░ 93.4%	468 / 501
early_access_features	███████████████████░ 93.8%	848 / 904
legal_documents	███████████████████░ 94.1%	1,568 / 1,667
endpoints	███████████████████░ 94.1%	8,606 / 9,143
messaging	███████████████████░ 94.3%	2,366 / 2,508
skills	███████████████████░ 94.4%	2,819 / 2,987
revenue_analytics	███████████████████░ 94.4%	3,586 / 3,797
data_catalog	███████████████████░ 94.6%	895 / 946
growth	███████████████████░ 94.9%	2,393 / 2,522
logs	███████████████████░ 95.3%	9,516 / 9,983
experiments	███████████████████░ 95.6%	24,055 / 25,164
replay_vision	███████████████████░ 95.6%	12,690 / 13,272
feature_flags	███████████████████░ 95.9%	14,351 / 14,964
warehouse_sources	███████████████████░ 96.1%	208,667 / 217,231
annotations	███████████████████░ 96.2%	732 / 761
user_interviews	███████████████████░ 96.4%	2,242 / 2,325
access_control	███████████████████░ 96.8%	849 / 877
customer_analytics	███████████████████░ 97.3%	7,277 / 7,481
analytics_platform	████████████████████ 98.2%	2,098 / 2,137
metrics	████████████████████ 98.3%	2,363 / 2,403
live_debugger	████████████████████ 99.2%	613 / 618
field_notes	████████████████████ 99.4%	158 / 159

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-13 19:16 UTC	Run
prod-us	✅ Deployed	2026-07-13 19:28 UTC	Run
prod-eu	✅ Deployed	2026-07-13 19:31 UTC	Run