feat(web-analytics): warm audience percentage via dagster run config - #70544
Conversation
lricoy
changed the title
|
Reviews (1): Last reviewed commit: "Merge branch 'master' into lricoy/wa-eag..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b14760950
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![stamphog[bot]](https://avatars.githubusercontent.com/in/3000332?s=60&v=4){kind=small}
There was a problem hiding this comment.
Opt-in-only change to an internal Dagster warming job (default behavior unchanged, scheduled runs unaffected); author is on the owning team with STRONG familiarity, and the two substantive review threads (Greptile, Codex) around validation, precompute gating, and warm-order priority were fixed and confirmed resolved in the current diff, with a positive Greptile reaction.
- Author wrote 100% of the modified lines and has 36 merged PRs in these paths (familiarity STRONG).
- 👍 on the PR from greptile-apps[bot], hex-security-app[bot].
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 93L, 1F substantive, 120L/2F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1c-medium (120L, 2F, single-area, feat) |
| stamphog 2.0.0b3 | .stamphog/policy.yml @ 1e39350 · reviewed head 3c2a974 |
1 participant
Problem
The eager warmer's audience is a static enrollment list (~88 teams), but ~8,900 teams open the web analytics dashboard daily. To decide whether warming should extend beyond the curated list, we want to scale the warm audience empirically and watch cluster behavior at each step.
Changes
active_teams_pctfield (0-100) on the warm job's Dagster run config, default 0 = current behavior (static enrollment lists only). The hourly schedule passes no config, so scheduled runs are unchanged.max_runtime.Operating it: launch the job manually from the Dagster UI with e.g.
ops: {warm_eager_baseline_op: {config: {active_teams_pct: 5}}}, watch aux insert counts / part+merge pressure / failure codes, then re-launch at 25, then 100. Stopping the experiment is just not launching again - the extra windows age out via TTL. Measured baseline for extrapolation: 88 teams ≈ 5.8k cold inserts / 8.5k insert-seconds, aux at ~13% CPU.How did you test this code?
Automatic notifications
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored with Claude Code. First iteration used a live instance setting; Lucas preferred a Dagster run-config parameter since he starts/stops the job from the UI anyway, so the knob moved onto the job itself with the default preserving the preselected list. Percentage semantics (rather than a raw team count) chosen per his framing - "a percentage of the active users on WA".