What about comparing test case runs over time? Is that a thing?

[carols10cents]

It's kind of a thing I've wanted Jenkins to do but haven't gotten it to do.

What I mean is: say we have 3 test cases and we've done 4 test runs in the last week--

	Test Run 1	Test Run 2	Test Run 3	Test Run 4
Test 1	F	P	P	P
Test 2	P	F	P	F
Test 3	P	P	B	B

F = failed, P = passed, B = blocked

So we could see that Test 1 was broken but now it's fixed, Test 2 is flaky and either the test or the functionality isn't consistently reliable, and something happened to block Test 3 two runs ago and is probably still going on.

Is this a thing that people do? Does it have a name? Is it worth talking about?

Maybe something like this will be in the "communicating with stakeholders" section. I feel like knowing the rate at which how many tests are going from failed to passed is something that could be useful to stakeholders to gauge completeness... ?

[laboon]

It is - I guess I would call it "test run tracking"? I'd probably also want to reference git bisect at this point. This will require a little bit of work/planning so I'm going to plan to work on it after the weekend.

This is definitely something I want the book to go into, and I discuss it in class. Thanks for bringing it up!