JUnit test XML does not include failures in non-declared test

[timothyg-stripe]

Description of the bug:

JUnit 4 runners like Mockito often fires test failures after a test has been done, as in here and here. Unfortunately, since the test description is dynamically created, in Bazel's JUnit test runner getTest(description) would return null. So the test failure is ignored and omitted from the JUnit XML file.

Ideally, Bazel's JUnit test runner should synthesize a test failure node, even for test cases that are not predeclared.

As an example, here's a sample test from our repository that uses MockitoJUnitRunner:

package com.stripe;

import org.junit.runner.RunWith;
import org.mockito.junit.MockitoJUnitRunner;

@RunWith(MockitoJUnitRunner.class)
public class BlahTest { ... }

Here's the form of the generated XML file:

<?xml version='1.0' encoding='UTF-8'?>
<testsuites>
  <testsuite name='com.stripe.BlahTest' timestamp='...' hostname='localhost' tests='10' failures='0' errors='0' time='6.662' package='' id='0'>
    <properties />
    <testcase name='test_...' classname='com.stripe.BlahTest' time='6.12' />
    ...
    <system-out />
    <system-err /></testsuite></testsuites>

Notice that failures and errors are both 0.

However, the test log shows one failure:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //src/test/java/com/stripe:BlahTest
-----------------------------------------------------------------------------
JUnit4 Test Runner
.OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

... logs redacted ...

E
Time: 6.793
There was 1 failure:
1) unnecessary Mockito stubbings(com.stripe.BlahTest)
org.mockito.exceptions.misusing.UnnecessaryStubbingException: 
Unnecessary stubbings detected in test class: BlahTest
Clean & maintainable test code requires zero unnecessary code.
Following stubbings are unnecessary (click to navigate to relevant line of code):
  1. -> at com.stripe.BlahTest.java:81)
Please remove unnecessary stubbings or use 'lenient' strictness. More info: javadoc for UnnecessaryStubbingException class.
	at org.mockito.internal.runners.StrictRunner.run(StrictRunner.java:52)
	at org.mockito.junit.MockitoJUnitRunner.run(MockitoJUnitRunner.java:163)
	at com.google.testing.junit.runner.internal.junit4.CancellableRequestFactory$CancellableRunner.run(CancellableRequestFactory.java:108)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at com.google.testing.junit.runner.junit4.JUnit4Runner.run(JUnit4Runner.java:116)
	at com.google.testing.junit.runner.BazelTestRunner.runTestsInSuite(BazelTestRunner.java:145)
	at com.google.testing.junit.runner.BazelTestRunner.main(BazelTestRunner.java:76)

FAILURES!!!
Tests run: 10,  Failures: 1


BazelTestRunner exiting with a return value of 1
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at ... --

Which category does this issue belong to?

Java Rules

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

See above for a sample test case. Unfortunately, I don't have a reproducible test case at hand.

Which operating system are you running Bazel on?

macOS

What is the output of bazel info release?

release 6.3.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response