fix(experiments): apply maturity filter to retention metrics#58970
Merged
Conversation
Contributor
|
🎭 Playwright didn't run on this PR — your changes touch code that could affect E2E behavior, but Playwright is opt-in via label now to keep CI cost down. Add the Most PRs don't need this. Real regressions still get caught on master and fix-forward. |
Contributor
|
Reviews (1): Last reviewed commit: "fix(experiments): apply maturity filter ..." | Re-trigger Greptile |
Contributor
|
Size Change: -5.51 kB (0%) Total Size: 118 MB 📦 View Changed
ℹ️ View Unchanged
|
andehen
approved these changes
May 19, 2026
Contributor
andehen
left a comment
andehen
left a comment
There was a problem hiding this comment.
Nice 👍 Simple change.
But, maturity is now counted from first exposure, not from the start event. So consider this example with a 7 day retention window:
- Day 1: user is exposed
- Day 20: user does the start event
- Day 21: experiment analysis runs
The user was marked mature on day 8 (exposure + 7d retention window), so they're included in the analysis. But the retention window from their start event is only 1 day in. We count them as "did not retain" when really they've barely had a chance.
So ideally we should count this from the start_event I think. I would assume that is what most users expect?
Fixing that is a larger change, but I think we should consider that in a follow-up PR. Would at least be good to point this out in the doc-string for future readers.
Contributor
Author
|
Hmm good point @andehen. Thanks for unblocking - I'll do this properly in a follow up. |
…on-maturity-filter # Conflicts: # products/replay_vision/frontend/generated/api.schemas.ts # services/mcp/src/api/generated.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The "Require completed conversion window" toggle does nothing for retention metrics. The feature only checks the
conversion_windowfield, but retention metrics store their time window inretention_window. So when you run a day-7 retention experiment, the denominator gets diluted by everyone who hasn't been around long enough to reach day 7.Changes
retention_window_endfor retention metrics, so the toggle works whether or not a conversion window is also set.How did you test this code?
Added
test_only_count_matured_users_without_conversion_windowcovering both the direct and precomputed exposure paths. All existing maturity tests across funnel, mean, ratio, and retention still pass.Publish to changelog?
no