KPI Request: Percent of Persistent Failures and Time Spent Fixing Failures

[chauhankaranraj]

As an OpenShift manager, I would like to see how many tests fail consistently and how much time is devoted to fixing failing tests, so that I can track our engineering efficiency over time.

On a job level, this analysis of failures can help understand the time and engineering resources spent on fixing issues (assuming more consecutive failures means more resources spent). A decrease in consecutive failures would suggest an improvement in the speed and efficiency of builds. A comparison can also be made between different jobs within the same dashboard to evaluate resources allocation.

How to collect metric
The testgrid platform records the result of each test run as one of the values specified in this doc. It also stores the timestamp at which each test was run. The metrics relevant to this issue can be calculated by finding cells values where “12”s are seen repeatedly, and looking at the corresponding timestamps.

Acceptance criteria

• small notebook in notebooks/data-sources/TestGrid/metrics/ that collects this metric and stores it in ceph as a parquet.

[MichaelClifford]

This notebook and existing function for finding consecutive failures might be helpful for this metrics.

[chauhankaranraj]

This notebook and existing function for finding consecutive failures might be helpful for this metrics.

Awesome, then I think we could just expand this function to calculate metrics like the average length (# cells), time difference, percent occurrence, etc :)

[chauhankaranraj]

Awesome, then I think we could just expand this function to calculate metrics like the average length (# cells), time difference, percent occurrence, etc :)

Turns out for many of the metrics @Shreyanand and I wanted to calculate, we can get away without actually unrolling the status dict. So since this function requires unrolled input, I think it won't be used in the notebook 😞

[MichaelClifford]

@chauhankaranraj that's great! not needing to unroll the data is probably better wherever possible