Skip to content

fix(cohorts): bound parity testing harness - #69816

Merged
matheus-vb merged 1 commit into
masterfrom
matheus-vb/parity-harness-fix
Jul 9, 2026
Merged

fix(cohorts): bound parity testing harness#69816
matheus-vb merged 1 commit into
masterfrom
matheus-vb/parity-harness-fix

Conversation

@matheus-vb

Copy link
Copy Markdown
Member

Problem

The compare_cohort_membership parity harness compares full converged snapshots, but the comparison is asymmetric.
The old side is always the complete snapshot while the new side only holds persons the pipeline has emitted for since the store wipe, so every untouched person counts as a divergence.
A cohort level warmup fast path hid this by declaring whole long window cohorts un gatable, so nothing meaningful actually gated.

Changes

This PR reworks how the existing harness diffs and gates membership.

  • The diff is now bounded to the observed universe O (persons the new pipeline actually decided on), since flip only emission means a never member emits nothing, so untouched old side persons are excluded instead of counted as divergences.
  • Within O the classifier removes expected skew (fresh and stale flips neither side has recomputed past yet) and gates the residual on a threshold.
  • To stop an O bounded gate from being fooled by a processor that emits nothing, a sampled probe checks the excluded persons (in old, never observed) for recent activity: active ones become suspect and gate FAIL where the behavioral window is covered, otherwise they report WARMUP, and the inactive rest are dormant.
  • WARMUP is now narrow (long window unobserved actives, reported not gated) rather than a blanket pass for whole cohorts.

How did you test this code?

90 pure unit tests, no services needed.
They pin the new fold observed helper and every classifier verdict path (excluded, fresh, stale, suspect FAIL on a covered window, WARMUP on a long one, the probe skip, the probe cutoff including the inf window overflow guard, the residual threshold, and the raw diff mode).

Docs update

No.

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 9, 2026 20:09
@posthog-project-board-bot posthog-project-board-bot Bot moved this to In Review in Feature Flags Jul 9, 2026
@matheus-vb matheus-vb added the stamphog Request AI approval (no full review) label Jul 9, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reworks an offline parity-testing/diagnostic management command (not production ingestion code, data models, or APIs), is well covered by 90 parameterized unit tests exercising the new logic paths, and the author is on the owning team with STRONG familiarity (100% of touched lines, all files previously modified).

  • Author wrote 100% of the modified lines and has 1 merged PRs in these paths (familiarity STRONG).
  • 👍 on the PR from hex-security-app[bot].
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list no deny categories matched
size 214L, 6F substantive, 433L/8F incl. docs/generated/snapshots — within ceiling
tier T1-agent / T1d-complex (433L, 8F, single-area, fix)
stamphog 2.0.0b3 .stamphog/policy.yml @ bf52e45 · reviewed head 6d9cfa9

@github-project-automation github-project-automation Bot moved this from In Review to Approved in Feature Flags Jul 9, 2026
@matheus-vb
matheus-vb enabled auto-merge (squash) July 9, 2026 20:23
@trunk-io

trunk-io Bot commented Jul 9, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@matheus-vb
matheus-vb merged commit c923800 into master Jul 9, 2026
351 checks passed
@matheus-vb
matheus-vb deleted the matheus-vb/parity-harness-fix branch July 9, 2026 20:46
@github-project-automation github-project-automation Bot moved this from Approved to Done in Feature Flags Jul 9, 2026
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 9, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-09 21:15 UTC Run
prod-us ✅ Deployed 2026-07-09 21:28 UTC Run
prod-eu ✅ Deployed 2026-07-09 21:30 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature/cohorts Feature Tag: Cohorts stamphog Request AI approval (no full review) team/feature-flags

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

1 participant