fix(experiments): add recalc queue-wait metric and adjust concurrency

[rodrigoi]

Problem

PostHog/charts#13049 moves the experiments recalculation Temporal worker from a static 2-pod pool to KEDA autoscaling driven by task queue backlog. Two gaps on the app side:

• The worker emits no schedule-to-start latency metric, so there's no worker-side signal for how long calc activities actually wait in the queue, which is the number that tells us whether the autoscaler is keeping up.
• The per-run metric fan-out (MAX_CONCURRENT_METRICS = 14) is misaligned with the per-org ClickHouse app-query concurrency limit (20) that governs recalc queries. Two concurrent runs from the same org demand 28 slots against a quota of 20, so the second run bounces off the limiter with ConcurrencyLimitExceeded instead of running at full width.

Changes

App-side counterpart to the autoscaling PR: adds the queue-pressure signal and aligns per-run concurrency with the org query quota.

Schedule-to-start latency histogram

New experiment_metrics_recalculation_activity_schedule_to_start_latency histogram recording scheduled-to-started time per activity, emitted from the existing metrics interceptor (mirrors the mcp_analytics intent clustering pattern). Buckets reach 30m, wider than the 5m execution-latency buckets, because queue wait under backlog is not bounded by the activity timeout.

• products/experiments/backend/temporal/recalculation_metrics.py
• posthog/temporal/common/worker.py

Per-run fan-out aligned to the org query limit

MAX_CONCURRENT_METRICS goes from 14 to 10 so two concurrent same-org runs fit exactly inside DEFAULT_APP_ORG_CONCURRENT_QUERIES (20). Pairs with the charts PR setting MAX_CONCURRENT_ACTIVITIES: 20 per pod, making one pod's slots equal one org's quota.

• products/experiments/backend/temporal/recalculation_workflow.py

How did you test this code?

pnpm turbo run backend:test --filter=@posthog/products-experiments

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Model: Fable 5
Manually refactored: yes

Skills used:

• /writing-pull-requests (local)

Relevant decisions:

• Verified which ClickHouse limiter actually governs recalc queries: they bypass the per-team API, dashboard, and materialized-endpoints limiters and are subject only to app_per_org (20), so the fan-out was sized against that number.
• The histogram gets its own bucket set instead of reusing the execution-latency buckets, since the interesting failure mode (queue wait growing past minutes under backlog) sits entirely above the 5m cap of the existing buckets.

[rodrigoi]

• fix(experiments): update error classification to prevent unnecessary retries #70915
• fix(experiments): add fairness key #70807
• fix(experiments): manage retries when CH concurrency limit is reached #70801
• fix(experiments): reverts retries to be idiomatic temporal #70793
• fix(experiments): add recalc queue-wait metric and adjust concurrency #69836 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[greptile-apps]

_{Reviews (1): Last reviewed commit: "fix(experiments): add missing obsrevabil..." | Re-trigger Greptile}

[github-actions]

🤖 CI report

⚠️ Backend coverage — 60.0% of changed backend lines covered — 2 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████░░░░░░░░ 60.0% (3 / 5)

File	Patch	Uncovered changed lines
products/experiments/backend/temporal/recalculation_metrics.py	50.0%	108–109

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29343953984 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
platform_features	██░░░░░░░░░░░░░░░░░░ 12.1%	7 / 58
batch_exports	████████░░░░░░░░░░░░ 39.7%	8,414 / 21,220
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
warehouse_sources_queue	████████████░░░░░░░░ 59.2%	148 / 250
tasks	█████████████░░░░░░░ 67.1%	25,151 / 37,465
data_tools	██████████████░░░░░░ 70.0%	63 / 90
ai_gateway	███████████████░░░░░ 75.0%	9 / 12
data_modeling	████████████████░░░░ 78.4%	4,696 / 5,987
signals	████████████████░░░░ 78.5%	18,103 / 23,059
cdp	████████████████░░░░ 80.7%	3,118 / 3,864
wizard	████████████████░░░░ 82.5%	772 / 936
cohorts	█████████████████░░░ 82.9%	3,048 / 3,675
notebooks	█████████████████░░░ 83.8%	6,086 / 7,259
agent_platform	█████████████████░░░ 84.1%	3,095 / 3,678
actions	█████████████████░░░ 86.6%	717 / 828
product_tours	█████████████████░░░ 87.5%	1,266 / 1,447
engineering_analytics	██████████████████░░ 87.6%	4,417 / 5,040
exports	██████████████████░░ 88.3%	6,857 / 7,763
visual_review	██████████████████░░ 88.5%	5,565 / 6,287
business_knowledge	██████████████████░░ 88.5%	4,400 / 4,969
conversations	██████████████████░░ 88.9%	15,924 / 17,921
mcp_analytics	██████████████████░░ 89.1%	2,485 / 2,790
dashboards	██████████████████░░ 89.1%	5,648 / 6,337
error_tracking	██████████████████░░ 89.5%	9,613 / 10,737
streamlit_apps	██████████████████░░ 90.4%	2,499 / 2,764
slack_app	██████████████████░░ 90.6%	9,460 / 10,444
links	██████████████████░░ 90.6%	183 / 202
marketing_analytics	██████████████████░░ 90.7%	11,476 / 12,646
product_analytics	██████████████████░░ 91.0%	5,452 / 5,991
managed_migrations	██████████████████░░ 91.9%	908 / 988
workflows	██████████████████░░ 92.0%	4,795 / 5,210
mcp_store	██████████████████░░ 92.1%	3,665 / 3,981
data_warehouse	██████████████████░░ 92.1%	17,357 / 18,843
alerts	██████████████████░░ 92.1%	3,389 / 3,678
web_analytics	███████████████████░ 92.7%	13,702 / 14,787
notifications	███████████████████░ 92.7%	1,026 / 1,107
ai_observability	███████████████████░ 92.7%	14,670 / 15,822
surveys	███████████████████░ 92.9%	5,660 / 6,094
posthog_ai	███████████████████░ 93.2%	1,311 / 1,407
tracing	███████████████████░ 93.2%	2,423 / 2,599
approvals	███████████████████░ 93.3%	3,395 / 3,640
reminders	███████████████████░ 93.4%	468 / 501
early_access_features	███████████████████░ 93.8%	848 / 904
legal_documents	███████████████████░ 94.1%	1,568 / 1,667
endpoints	███████████████████░ 94.1%	8,606 / 9,143
skills	███████████████████░ 94.4%	2,827 / 2,995
revenue_analytics	███████████████████░ 94.4%	3,586 / 3,797
messaging	███████████████████░ 94.5%	2,530 / 2,677
review_hog	███████████████████░ 94.5%	6,429 / 6,802
growth	███████████████████░ 94.9%	2,393 / 2,522
logs	███████████████████░ 95.3%	9,441 / 9,907
experiments	███████████████████░ 95.6%	24,047 / 25,156
replay_vision	███████████████████░ 95.6%	12,690 / 13,272
feature_flags	███████████████████░ 96.0%	14,600 / 15,203
warehouse_sources	███████████████████░ 96.1%	214,193 / 222,893
annotations	███████████████████░ 96.2%	732 / 761
user_interviews	███████████████████░ 96.4%	2,242 / 2,325
access_control	███████████████████░ 96.8%	849 / 877
data_catalog	███████████████████░ 97.2%	1,642 / 1,689
customer_analytics	███████████████████░ 97.3%	7,396 / 7,600
analytics_platform	████████████████████ 98.2%	2,098 / 2,137
metrics	████████████████████ 98.3%	2,363 / 2,403
live_debugger	████████████████████ 99.2%	613 / 618
field_notes	████████████████████ 99.4%	158 / 159

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-15 17:16 UTC	Run
prod-us	✅ Deployed	2026-07-15 17:35 UTC	Run
prod-eu	✅ Deployed	2026-07-15 17:33 UTC	Run