Mismatch between test summary and tests reported

[mrginglymus]

Having used the ability to have custom checks name in anger, I spotted what I thought was a missing feature, but actually turns out to be appear to be a bug, possibly on this line:

junit-plugin/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java

Line 287 in d3fd50c

new JUnitChecksPublisher(action, summary).publishChecks(listener);

as the action here is a merge of all junit invocations in a run, but the summary is only the count of passes/fails in the current invocation of junit.

For example:

stage('first stage') {
    junit 'test1.xml' // contains 1 failure, test_foo
}
stage('second stage') {
    junit 'test2.xml' // contains 2 failures, test_bar and test_baz
}

Will result in a check being published that will claim that there are only two failures, but list all three of test_foo, test_bar and test_baz.

The behaviour that I would find most useful is to only report in the checks the failures in that current invocation; the fix for this looks simple enough as far as I can tell, and I'll try to put together a PR if that's the option that's desirable.

It's also entirely possible I've not understood the code and not spotted the actual cause, but the effect definitely appears to be there:

[timja]

Sounds right to me

[jglick]

only report in the checks the failures in that current invocation

Would it not be preferable to list all the failures and the aggregate count? Why would you want to overwrite a Check with less information? Context: #210 (comment)