Statistical Estimates

Statistical Estimates

In this section, we will explain three useful outputs of using this tool. While the visualizations are interesting, they do not numerically show how much your product is improving your KPIs.

Please look at the image of the command line below, as I describe the output line by line. I will be very brief about what each number means. Further detail can be found by following the links in the "More Information" section.

1. Difference-In-Difference Estimate with Confidence Estimate
   1. 'estimate' estimates the size of the effect of treatment on the treated. This number is measured in the units of the provided KPI.
   2. 'confidence interval' is a 95% confidence interval for the effect. It's often nice to have an interval to estimate a parameter for the whole population, rather than just providing a number, because there is uncertainty in our estimate.
2. Average Improvement in Treatment Group
   1. 'test statistic' provides the test statistic for running a 2-sided 2-sample T-test for this question.
   2. 'p-value' provides the probability of generating our test statistic or a more extreme test statistic assuming our null hypothesis and model is true. This will be a value between 0 and 1.
   3. 'estimate' estimates the size of the effect of the mean difference (across identifiers/users) between the mean daily KPI for developer before and after treatment. This value is in the units of the provided KPI.
3. Total ROI for Treatment Group
   1. 'test statistic' provides the test statistic for running a 2-sided 2-sample T-test.
   2. 'p-value' provides the probability of generating our test statistic or a more extreme test statistic assuming our null hypothesis and model is true. This will be a value between 0 and 1.
   3. 'estimate' estimates the size of the effect of the mean difference (across identifiers/users) between the summed daily KPI for developer before and after treatment. This value is in the units of the provided KPI.