Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust CIs when adjusting pvalues #1740

Merged
merged 18 commits into from Oct 13, 2023
Merged

Adjust CIs when adjusting pvalues #1740

merged 18 commits into from Oct 13, 2023

Conversation

lukesonnet
Copy link
Contributor

@lukesonnet lukesonnet commented Oct 11, 2023

Features

Right now we don't update our CIs when we apply multiple comparisons corrections.

There isn't a great way to do this for all tests because:

  1. Existing methods (Benjamini & Yeukateli 2005) only apply to selected parameters
  2. Ad-hoc methods (finding a std-dev that matches to the adjusted "p"-value) don't handle the common case where adjusted "p"-values are 1.

See the thread on this here: https://linen.growthbook.io/t/13221597/hi-u01tcpdb58c-u01t6hchd0a-and-u03gc3g9e8m-we-re-using-the-e

Nonetheless, having CIs out of sync with the p-values is disorienting and a bad user experience.

So we implemented a variant of the ad-hoc method (item 2 above) to adjust CIs when an org implements multiple comparisons corrections:

  1. For all adjusted p-values, compute the implied stddev that would have yielded that p-value (note we use a z test in the app because only the stats engine has access to the true DoF we used for the t test; this should not matter for this method but could be improved). If the p-value is > 0.999999, we set the CI to be infinite.
  2. Using that implied stddev, compute new CIs
  3. If the new CIs are wider than the old CIs; overwrite the old CIs in the front-end.

@github-actions
Copy link

github-actions bot commented Oct 11, 2023

Your preview environment pr-1740-bttf has been deployed.

Preview environment endpoints are available at:

@github-actions
Copy link

github-actions bot commented Oct 13, 2023

Deploy preview for docs ready!

✅ Preview
https://docs-akinyz7xg-growthbook.vercel.app

Built with commit a46dfb5.
This pull request is being automatically deployed with vercel-action

@lukesonnet lukesonnet marked this pull request as ready for review October 13, 2023 20:58
@lukesonnet lukesonnet merged commit cccf45b into main Oct 13, 2023
6 checks passed
@lukesonnet lukesonnet deleted the ls/adjust-cis branch October 13, 2023 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants