Feature: Report time-quantiles

[caje731]

In the generated report, I currently see metrics such as these:

Average time to first response	22:26:39.529412
Average time to close	3 days, 17:21:22.066667
Average time to answer	None

While averages are helpful, they can be biased by outliers. For example, a PR that took forever to close skews the reported average for the entire team, and it would be an unfair representation of how the team works. Skews occur a lot more than one may imagine, so I propose to introduce quantiles - specifically the median, P80, P95, P99.

Better yet, configuration-driven time metrics, letting the user choose what kind of time-reporting they need.

[zkoppert]

Super cool idea! Thanks for opening up the issue. I think adding a configuration option for this would be best so we don't have to change existing functionality out from under users. Something like REPORT_QUANTILES=True

[caje731]

Might want to keep it flexible a bit, like so: REPORT_QUANTILES=10[,20,30]. So the freedom to "bring your own stripes" instead of shoe-horning everyone into the same segments.

So folks that want to focus on just the median can get a more concise report with REPORT_QUANTILES=50 than people that want to focus on how the overall trend looks like, with REPORT_QUANTILES=50,80,95,99

We may also want to note that at least 3 actions for which the averages are reported - Time to discuss, time to close, time to first response - [may be more]. So today for M actions, we have M times reported [all averages], but if one introduces N quantiles like these, they will get M * (N + 1) times in the report. So probably good to think about "presentation" too.

[MaineC]

For implementing the quantiles computation - likely this would need adding numpy to the dependencies or am I missing something simpler?

@caje731 First response, close and answer really seem to be the only locations where averages are computed at the moment.

[spier]

A first version of this idea has been implemented by the amazing @MaineC in #127.

@caje731 does this go into the direction that you had in mind? The stats now look like this:

[caje731]

It is a great start! Thank you for working on it.
You can elevate my happiness to bliss with configurable quantiles (as in our comments above), but this is already promising ✌️

[github-actions]

This issue is stale because it has been open 21 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stalled for 35 days with no activity.