How can an extension add to the output of a test?

[toolforger]

I am writing an extension to inject java.util.Random objects as parameters for test methods, and things are progressing well.

The problem I am facing is that the extension has to report the random seed in case of test failure.
I tried various approaches (logging, TestWatcher and either add a suppressed exception to the Throwable or publishing a report entry, AfterTestExecutionCallback), but each would eventually run into a problem that I don't know how to solve, so I'm asking here.

Important goals:

• Should not require additional configuration parameters to work. (Likely rules out logging.)
• The output should be firmly associated with the failing test, so that test reporting tools keep that output with the test's failure message.
• The output should really be part of what Gradle considers "Failure Details". ExtensionContext.publishReportEntry did nothing, System.out.println would actually work but make Gradle separate out the stdout output to a separate tab, and in IntelliJ the output would come before the failure report.

Am I trying something impossible, or am I missing the intended approach?

[vshulcz]

The piece you're missing is TestExecutionExceptionHandler. It runs while the failure is still in flight, which is the only point where you can attach something that ends up inside the failure itself rather than beside it.

The contract is explicit about what you may do with the throwable:

Implementors must perform one of the following.
1. Swallow the supplied throwable, thereby preventing propagation.
2. Rethrow the supplied throwable as is.
3. Throw a new exception, potentially wrapping the supplied throwable.

Option 2 with a suppressed exception attached is what fits your goal, since the seed then travels with the failure rather than as separate output:

class RandomSeedExtension implements ParameterResolver, TestExecutionExceptionHandler {

    private static final Namespace NS = Namespace.create(RandomSeedExtension.class);

    @Override
    public Object resolveParameter(ParameterContext pc, ExtensionContext ctx) {
        long seed = ...;
        ctx.getStore(NS).put("seed", seed);
        return new Random(seed);
    }

    @Override
    public void handleTestExecutionException(ExtensionContext ctx, Throwable t) throws Throwable {
        Long seed = ctx.getStore(NS).get("seed", Long.class);
        if (seed != null) {
            t.addSuppressed(new IllegalStateException("random seed = " + seed));
        }
        throw t;
    }
}

Suppressed exceptions are printed as part of the stack trace, so Gradle keeps them in the failure details and IntelliJ shows them under the failure rather than in a separate stdout tab. That covers the ordering problem you hit with System.out.println.

Two related notes. If the seed also matters when a failure comes from @BeforeEach or @AfterEach, implement LifecycleMethodExecutionExceptionHandler too, since those exceptions never reach the test handler. And storing the seed in the extension context store, rather than a field, keeps it correct for parallel execution.

publishReportEntry did nothing visible for a reason: report entries go to the reporting stream, not into the failure, so Gradle's console output ignores them even though they show up in the XML/HTML reports.

[toolforger]

Adding a suppressed exception seems like an abuse to me, because:

1. The exception will get printed (good), but it will be printed with additional noise in the form of a stack trace.
2. If somebody wants to attach some output in the non-failure case, an exception won't work.
   I actually have a use case: There's a test that takes one of two paths, depending on System.nanoTime() granularity.

But ah well if a suppressed exception is the one way to deal with that, then so be it.

I'm considering workarounds for the two above artifacts:

1. Make my own exception class that prints just the message and nothing else.
   I'll have to take a closer look to see what actually needs to be overridden, or if things can even be overridden; if that doesn't work, then ah well.
   Adding that exception class seems like a lot of ad-hockery to me. Like I'm putting a band-aid over an architectural issues. So I fully expect to undo that should JUnit ever offer a cleaner way to pass up additional information. For now, I think it's good enough
2. For output in the success case, I think I'll just print to stdout and call it a day.

Comments, perspectives, criticism, feedback welcome :-)

(No need to deal with @Foreach etc., this is all just for test method parameters. Fortunately not all potential complications materialized, that's nice.)

[toolforger]

Works as designed.

Gradle output:

> Task :randomness:jar

Expected :null
Actual   :java.util.Random@192f2f27
<Click to see difference>

org.opentest4j.AssertionFailedError: 
expected: null
 but was: java.util.Random@192f2f27
	at org.toolforger.junit.randomness.RandomnessExtensionTest.failingTest(RandomnessExtensionTest.java:59)
	Suppressed: org.toolforger.junit.randomness.Info: 
		Attached info message


> Task :randomness:test FAILED

No idea why Gradle is adding these copious amounts of blank lines, but I seem to recall that it has always been that way.

The Gradle report looks exactly as it should:

org.opentest4j.AssertionFailedError: 
expected: null
 but was: java.util.Random@493dfb8e
	at org.toolforger.junit.randomness.RandomnessExtensionTest.failingTest(RandomnessExtensionTest.java:59)
	Suppressed: org.toolforger.junit.randomness.Info: 
		Attached info message

So - all is fine, except I still find it annoying to have to write that org.toolforger.junit.randomness.Info (and everybody else who wants to attach infos to exceptions will have to write it as well).
Heh. Maybe I should publish that thing as org.toolforger.exceptions.Info.

[vshulcz]

You don't need to override much. Throwable has a constructor for exactly this, and printStackTrace prints the result of toString() as its header:

public final class Info extends RuntimeException {
    public Info(String message) {
        super(message, null, false, false); // no suppression, no writable stack trace
    }

    @Override
    public String toString() {
        return getMessage();
    }
}

That gets you Suppressed: Attached info message with no class name and no frames.

On the success case, publishReportEntry isn't doing nothing, Gradle just doesn't render it. XmlReportWriter folds report entries into the <system-out> element of the JUnit XML, so they do survive into anything that reads those files. If your output only needs to be visible in CI rather than in Gradle's console, that path works without stdout.

[toolforger]

I'm doing exactly that; that code is finished :-)
I'm extending Exception though, just to make it a checked exception.
It's just a roadbump to discourage developers from actually throwing it.

In case you're interested, here's the full code.
As usual, the amount of stone turning is more than meets the eye, but here we go:

package org.toolforger.junit.randomness;

import java.util.Objects;

/**
 * This exception is supposed to be added as a suppressed exception to existing {@link Throwable}s,
 * and not intended to be thrown on its own as it does not carry a stack trace or suppressed exceptions.
 * <p>
 * Note 1: The class name does not carry the customary {@code Exception} in its name to underline that it is not really
 * an exception.<br>
 * This results in a stack trace output like this:
 * <pre>
 * org.opentest4j.AssertionFailedError:
 * expected: null
 * but was : java.util.Random@3bffddff
 *         at &lt;package.name&gt;.&lt;className&gt;.&lt;methodName&gt;(&lt;className&gt;.java:59)
 *         ... &lt;possibly more stack trace lines&gt;
 *         Suppressed: org.toolforger.junit.randomness.Info:
 *                 &lt;message from constructor&gt;
 * </pre>
 * <p>
 * Note 2: Suppressed exceptions exist since Java 7, so this code will not make sense in earlier Java
 * (and possibly not even compile).
 * <p>
 * Note 3: This class extends {@link Exception} to make it a checked exception.<br>
 * There is no semantic reasoning here, it's just to help people avoid accidentally throwing an {@code Info} object.
 */
public class Info extends Exception {

  /**
   * The sole constructor.
   * <p>
   * It does not allow a cause, suppressed exceptions, or a stack trace.
   */
  public Info(String message) {
    // Throwable.printStackTrace uses System.lineSeparator() and "\t" for formatting,
    // so we use the exact same mechanisms to avoid any formatting surprises.
    // After all, even if it's just printed, tools that read stack traces may get confused if we do something unusual.
    super(System.lineSeparator() + "\t\t" + message, null, false, false);
  }

  @Override
  public boolean equals(Object other) {
    // For tests that wish to check that the correct information is added.
    return other.getClass() == Info.class && Objects.equals(getMessage(), ((Info) other).getMessage());
  }

  @Override
  public int hashCode() {
    return Objects.hashCode(getMessage());
  }
}

[toolforger]

BTW yeah console output is important.
Having something show up just in the XML files and the reports generated from them just means more questions from console users, and that's a support overhead to be avoided if possible.

Actually, now that I think of it, I have a strong feeling that nobody should use the reporting API for any data that's for human consumption. Data that's exclusively consumed by machines, such as formatting hints, raw statistic data, and similar, seem fine to me.

[vshulcz]

Console visibility is the right thing to optimise for, but it's worth pinning the blame precisely: the platform does render report entries for humans, Gradle just doesn't.

TreePrintingListener and FlatPrintingListener in junit-platform-console both implement reportingEntryPublished and hang the entry off the test node, so with --details=tree or --details=flat the entries print under the test they belong to. There's a fileEntryPublished next to it for the newer publishFile path. Gradle's test event model has no slot for either, so they're dropped before anything could show them.

So the API isn't machine-only by design. For an extension shipped to Gradle users the practical outcome is the same, which is why the suppressed-exception route stays the better call here.