Skip to content

feat(web-analytics): make warm team concurrency launch-configurable - #70617

Merged
lricoy merged 2 commits into
masterfrom
lricoy/wa-warm-concurrency-config
Jul 14, 2026
Merged

feat(web-analytics): make warm team concurrency launch-configurable#70617
lricoy merged 2 commits into
masterfrom
lricoy/wa-warm-concurrency-config

Conversation

@lricoy

@lricoy lricoy commented Jul 14, 2026

Copy link
Copy Markdown
Member

Problem

The eager warm/backfill op runs teams through a fixed ThreadPoolExecutor(max_workers=10). During the fleet backfill we may want to raise throughput when the offline tier (where the insert SELECTs execute) is quiet - but 10 is compiled in, so any experiment needs a deploy.

Changes

  • New team_concurrency run-config field (default 10 = current behavior, capped at 25). The scheduled hourly job passes no config and is unchanged; manual launches (including web_analytics_eager_backfill) can override it.
  • The cap is deliberately conservative: the offline tier is shared (cohorts, exports, experiments) and its binding constraint is concurrency slots (202 TOO_MANY_SIMULTANEOUS_QUERIES), not CPU. The field's docstring carries the operating guidance: only raise when offline avg CPU is quiet (<~40%), watch for 202s.

Measured context: at 10 workers the backfill sustains ~12-27k inserts/hr with zero failures; tonight's offline tier sat at 40-77% CPU with one node over its core count, which is exactly the condition under which this knob should NOT be raised.

How did you test this code?

  • Mechanical change (constant -> validated config field with the constant as default); the op's existing tests exercise the pool path via direct invocation with default config. Field validation is pydantic (ge=1, le=25).

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Authored with Claude Code during the live fleet backfill. Lucas asked whether increased concurrency was affordable; measurement said not right now (offline tier at 40-77% CPU, one node saturated by other tenants), so the knob ships with a conservative cap and explicit guidance instead of a raised default.

@lricoy lricoy self-assigned this Jul 14, 2026
@lricoy
lricoy marked this pull request as ready for review July 14, 2026 03:10
@lricoy lricoy added the stamphog Request AI approval (no full review) label Jul 14, 2026
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 14, 2026 03:11
@greptile-apps

greptile-apps Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "feat(web-analytics): make warm team conc..." | Re-trigger Greptile

stamphog[bot]
stamphog Bot previously approved these changes Jul 14, 2026

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trivial, well-scoped config change by a STRONG-familiarity author on the owning team, with positive bot reactions and no outstanding concerns.

  • Author wrote 100% of the modified lines and has 38 merged PRs in these paths (familiarity STRONG).
  • 👍 on the PR from chatgpt-codex-connector[bot], greptile-apps[bot], hex-security-app[bot].
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list no deny categories matched
size 7L, 1F substantive — within ceiling
tier T1-agent / T1a-trivial (7L, 1F, single-area, feat)
stamphog 2.0.0b3 .stamphog/policy.yml @ 14c26ac · reviewed head 75b00b4

@lricoy
lricoy enabled auto-merge (squash) July 14, 2026 03:24
@stamphog
stamphog Bot dismissed their stale review July 14, 2026 03:27

New commits pushed (delta classified non_trivial_delta) — stamphog approval dismissed; re-review running automatically.

@trunk-io

trunk-io Bot commented Jul 14, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trivial, well-scoped config knob with a validated cap and unchanged default; author owns the code with strong familiarity and no outstanding concerns.

  • Author wrote 100% of the modified lines and has 38 merged PRs in these paths (familiarity STRONG).
  • 👍 on the PR from chatgpt-codex-connector[bot], greptile-apps[bot], hex-security-app[bot].
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list no deny categories matched
size 9L, 1F substantive — within ceiling
tier T1-agent / T1a-trivial (9L, 1F, single-area, feat)
stamphog 2.0.0b3 .stamphog/policy.yml @ 14c26ac · reviewed head bd261d7

@lricoy
lricoy merged commit bcea1e0 into master Jul 14, 2026
177 checks passed
@lricoy
lricoy deleted the lricoy/wa-warm-concurrency-config branch July 14, 2026 03:37
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 14, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-14 04:01 UTC Run
prod-us ✅ Deployed 2026-07-14 04:11 UTC Run
prod-eu ✅ Deployed 2026-07-14 04:12 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stamphog Request AI approval (no full review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant