Description
Rarely, a test has a non-OK harness status, but passes some number of subtests. During aggregation, this partial failure is not displayed correctly.
-
NOTE: The harness status is NEVER counted toward the score on the Interop Dashboard, which is why the discrepancy arises. The searchcache that populates the results page is stored in a way to quickly reference and aggregate subsets of test run data. See the results analysis script for information on interop score aggregation.
-
When the searchcache aggregates subtest scores, it filters out the harness status OK scores, which are stored as subtests results themselves. "OK" results are unique to the harness status result, so they can be filtered out and not counted toward the test score. Harness results that are non-passing have the same statuses as non-harness results, e.g. "TIMEOUT", "CRASH", etc, so the searchcache aggregation does not know which subtest result is the harness status.
Example:
This test has a harness result TIMEOUT for the browser in the right-most column. When looking at the single test view, the total displays "8/9" subtests passing, which is not counting the harness result TIMEOUT (correctly). The error occurs when looking at a view higher than the single test view.
Here, we can see that the searchcache did not differentiate the harness result from the other subtest results, so the ratio displays an 8/10 instead of 8/9. This scenario happens rather rarely, but has come up before.
The likely best option for fixing this problem is to add some way to discern the harness result as being the harness result in the data stored that is aggregated by the search cache rather than only filtering "OK" statuses.