Skip to content

Commit

Permalink
Fix experiment-results docs from bayesian update (#2514)
Browse files Browse the repository at this point in the history
  • Loading branch information
lukesonnet committed May 13, 2024
1 parent 1bfa804 commit 8e56283
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
2 changes: 1 addition & 1 deletion docs/docs/experimentation-analysis/experiment-results.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ A "difference type" is the way we measure the difference in variations. There ar
- `Absolute` - This is simply the difference in average values across your variations --- $\mu_t - \mu_c$. This can help you understand the raw difference in average values, e.g. the treatment leads to an increase in revenue of \$0.31 per user in the above example.
- `Scaled Impact` - This helps you understand the daily total (as opposed to average) effect that your experiment would have had if 100% of users that would have been exposed to your treatment variation had gotten that treatment variation. It is computed as $(\mu_t - \mu_c) * (N_t / p_t) * (1 / d)$, where $N_t$ is the number of users that are in your treatment variation, $p_t$ is the percent of all traffic that is in that variation, and $d$ is the number of days in the current phase used for the results. So if your experiment ran on 10% of traffic for 20 days, with 5% going to the treatment variation, and there were 5,000 users in your treatment variation, the scaled impact would be $0.31 * 5,000 / 0.05 / 20 = 1,550$ dollars per day. This implies that this experiment would have lead to a $1,550$ increase in total revenue per day if every user that could have been exposed the variation had been exposed to the variation.

These difference types can have slightly different statistics because for Relative effects we need to account for the uncertainty in estimating $\mu_c$ (which forces us to use a delta method derived variance to properly handle). For more details, see the [Statistical Details](statistics/statistics-details).
These difference types can have slightly different statistics because for Relative effects we need to account for the uncertainty in estimating $\mu_c$ (which forces us to use a delta method derived variance to properly handle). For more details, see the [Statistical Details](/statistics/details).

Furthermore, with CUPED enabled, you may find that the changes are not exactly the same as the difference in raw variation averages, due to CUPED adjusting those averages under the hood to reduce variance.

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/statistics/details.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -189,4 +189,4 @@ where $M_i$ and $D_i$ are the $i^{\text{th}}$ units' values for the numerator an

### Quantile metrics

The statistics for quantile metrics are covered more in detail in the [Quantile documentation](/statistics/quantile). But in the end we arrive at both a `\hat\mu` and `\hat\sigma^2` for the desired quantile and its variance and use those in our lift calculations.
The statistics for quantile metrics are covered more in detail in the [Quantile documentation](/statistics/quantile). But in the end we arrive at both a $\hat\mu$ and $\hat\sigma^2$ for the desired quantile and its variance and use those in our lift calculations.

1 comment on commit 8e56283

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for docs ready!

✅ Preview
https://docs-mh2lddww8-growthbook.vercel.app

Built with commit 8e56283.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.