Skip to content

feat(workflows): size email reputation windows to each sender's own volume - #73465

Merged
meikelmosby merged 1 commit into
masterfrom
posthog-code/workflow-reputation-representative-volume
Jul 24, 2026
Merged

feat(workflows): size email reputation windows to each sender's own volume#73465
meikelmosby merged 1 commit into
masterfrom
posthog-code/workflow-reputation-representative-volume

Conversation

@meikelmosby

Copy link
Copy Markdown
Collaborator

Problem

The reputation evaluator's window is "at least the last 24h AND at least the last 1,000 sends". The fixed 1,000 is below a single campaign for most workflow senders, which creates a washout: send a 1,000-email batch at 50% hard bounce on Monday, send one clean 1,000-email batch on Tuesday, and Tuesday's snapshot reports 0% / healthy – Monday's disaster fell entirely out of the window. That's a false all-clear on the exact signal enforcement decisions (Slack alerts, manual suspension) are built on, and it's trivially gameable: one clean blast launders any reputation.

AWS SES – whose account-level view this score exists to predict – computes rates over "representative volume" sized to the sender's own scale, explicitly to "limit the influence of any single email campaign". Our fixed floor captured the volume-based idea but missed the per-sender sizing that gives it teeth. AWS still remembers Monday on Tuesday; we shouldn't forget it either.

How it works now

Each target's window volume is sized from its own sending scale:

representative volume = max(TARGET_VOLUME, VOLUME_MULTIPLIER × biggest sending day in the lookback)

with VOLUME_MULTIPLIER = 3 and TARGET_VOLUME = 1000 (both env-tunable, no schema or query changes – the biggest day is computed from the hourly buckets the evaluator already has in memory). The window walk itself is unchanged: everything in the last 24h unconditionally, then older hourly buckets until the volume is met, capped at 30 days.

Anchoring on the biggest single day rather than an average is deliberate: a weekly 10k-batch sender averages ~1.4k/day – less than one campaign, so average-based sizing would collapse their window – but their max day is 10k, so their window spans their last 3 batches. The multiplier is the knob with a plain-English meaning: no single day can be more than 1/3 of the window, and redeeming a bad campaign takes 3 clean ones.

Worked examples (all with hard-bounce thresholds 2% warning / 5% critical):

Sender Window The washout day, replayed
1k/day batch sender 3k sends ≈ 3 days Day 1 @50% → 50% critical. Day 2 clean → 25% critical (was: 0% healthy). Day 3 → 16.7%. Day 4 → healthy. Decay by dilution, not amnesia.
Weekly 10k batches 30k ≈ 3 Mondays A 6% Monday stays critical through the silent week, 3% warning after one clean batch, healthy after three.
1M/day 3M ≈ 3 days A 100k blast @50% shows as ~1.7% for 3 days – visible, not diluted into a month of volume.
500/month floor (1k) → 30d cap Unchanged from today.

Known trade-off (accepted, not fixed here): a sender who was big last month and is tiny now keeps a large representative volume until the big days age out of the 30-day lookback, so their score reacts slowly to fresh problems. A gone-quiet sender is low-risk to the shared account by definition; if it matters in practice the fix is decaying the max-day lookback, which is a follow-up knob.

Changes

  • representativeVolume() in the evaluator service + both call sites (per-workflow and per-team windows)
  • EMAIL_REPUTATION_VOLUME_MULTIPLIER env config (default 3), wired through the Temporal worker
  • Existing window-walk tests pinned unchanged by running them at multiplier 1 (identity); new test block covers the sizing function (floor, multiplier, max-vs-average anchoring, empty/bounce-only edge cases) and the two scenario regressions that motivated this: the one-clean-batch washout and the weekly-batch-sender window

Note

Rates can shift on deploy: senders whose windows now reach further back may see higher (more honest) rates and some healthy→warning transitions. Since the score is still visibility-only and #73438's notifications key off transitions, expect a small burst of warning notifications on the first post-deploy evaluation if those land together.

How did you test this code?

  • 14/14 evaluator tests pass locally: all 8 pre-existing window-walk tests unchanged (multiplier 1 pins them exactly), 5 new sizing unit cases, 2 new end-to-end scenario tests (washout prevention at 25% critical; weekly batch sender at 3% warning)
  • nodejs typecheck clean for changed files (remaining errors are pre-existing on master in unrelated files)

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Built with PostHog Code (Claude). Skills invoked: /writing-tests. Design came out of a scenario-analysis session: the fixed-window washout was found by walking a two-batch example, then three alternatives were compared (raise the time floor, fixed 30-day sliding window, per-sender representative volume). The 30-day window fixes the washout but mutes catastrophic days for high-volume senders (50k bounces / 30M sends = 0.17%) and makes redemption calendar-based; per-sender sizing handles both ends proportionately. AWS doesn't publish its formula – m × max_daily is our own construction of their documented principle, chosen because the max-day anchor provably bounds any single day to 1/m of the window. The declining-sender refinement (decayed max) was considered and deliberately deferred. Note: the window tooltip added in #73437 describes the old fixed-1,000 window – whichever of these merges second should reword it to "about 3× your biggest sending day, minimum 1,000 emails".


Created with PostHog Code

…olume

Generated-By: PostHog Code
Task-Id: 0f3e9835-ed9b-469d-aa52-1152818572a2
@meikelmosby meikelmosby self-assigned this Jul 24, 2026
@meikelmosby meikelmosby added the reviewhog ($$$) Reviews pull requests before humans do label Jul 24, 2026
@posthog

posthog Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

🦔 ReviewHog reviewed this pull request

Nothing worth raising this time, so here's a calming picture instead:

A happy dog on a sunny path

@meikelmosby
meikelmosby marked this pull request as ready for review July 24, 2026 12:58
@meikelmosby
meikelmosby requested a review from a team July 24, 2026 12:58
@greptile-apps

greptile-apps Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
nodejs/src/cdp/config.ts:358
**Deployment override remains unavailable**

The new setting is described as environment-tunable, but production deployment configuration does not expose it through posthog/charts and posthog/secrets, so operators remain fixed to the default multiplier until that deployment wiring is added.

Reviews (1): Last reviewed commit: "feat(workflows): size email reputation w..." | Re-trigger Greptile

Comment thread nodejs/src/cdp/config.ts
// ships separately.
EMAIL_REPUTATION_EVALUATION_HOUR_UTC: 6,
EMAIL_REPUTATION_TARGET_VOLUME: 1000,
EMAIL_REPUTATION_VOLUME_MULTIPLIER: 3,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Deployment override remains unavailable

The new setting is described as environment-tunable, but production deployment configuration does not expose it through posthog/charts and posthog/secrets, so operators remain fixed to the default multiplier until that deployment wiring is added.

Rule Used: When a new secret or environment variable is added... (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: nodejs/src/cdp/config.ts
Line: 358

Comment:
**Deployment override remains unavailable**

The new setting is described as environment-tunable, but production deployment configuration does not expose it through posthog/charts and posthog/secrets, so operators remain fixed to the default multiplier until that deployment wiring is added.

**Rule Used:** When a new secret or environment variable is added... ([source](https://app.greptile.com/posthog-org-19734/github/PostHog/posthog/-/custom-context?memory=c7932f06-eec7-4fbe-b35f-ac93bc7b7a80))

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@mayteio mayteio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📧 ✅

@meikelmosby
meikelmosby merged commit 651026a into master Jul 24, 2026
351 of 355 checks passed
@meikelmosby
meikelmosby deleted the posthog-code/workflow-reputation-representative-volume branch July 24, 2026 13:54
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 24, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-24 14:42 UTC Run
prod-us ✅ Deployed 2026-07-24 14:58 UTC Run
prod-eu ✅ Deployed 2026-07-24 15:01 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

reviewhog ($$$) Reviews pull requests before humans do

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants