Bug 1212873 - Compute the 95% confidence interval for mean #27
Conversation
|
|
||
| statistics[contextKey].push({ | ||
| Metric: key, | ||
| Mean: mean, | ||
| Median: stats.median(values), | ||
| Min: Math.min.apply(Math, values), | ||
| '95th Perc.': stats.percentile(values, 0.95) || mean, |
There was a problem hiding this comment.
How about making this just 95%?
There was a problem hiding this comment.
Wouldn't that be easily confused with the 95% confidence interval? How about we just remove this altogether?
There was a problem hiding this comment.
The tough thing is the current dependency with how things are done right now, with the way things should actually be. For example, right now all of our metrics in the dashboards display p95. All of our baselines for v2.2 are calculated using p95. All our regression detection scripts query for p95.
All that is to say: we could easily switch everything over to use mean, but we would have to change the messaging around our releases to say that we are abandoning p95 for another alternative. Thoughts?
FYI, here is all the stats for v2.2 I gathered. https://gist.github.com/eliperelman/5aac3b94e2f815dbb827
There was a problem hiding this comment.
OK, remove p95, use 95% Bound instead.
There was a problem hiding this comment.
I'll merge this, remove p95, then publish as Raptor v3.0.
Bug 1212873 - Compute the 95% confidence interval for mean
https://bugzilla.mozilla.org/show_bug.cgi?id=1212873