Add dedicated timeout parameter for `@FuzzTest`

[Marcono1234]

Problem

JUnit version: 5.13.0
Jazzer JUnit version: 0.24.0

Currently Jazzer recognizes JUnit's @Timeout annotation, see

jazzer/src/main/java/com/code_intelligence/jazzer/junit/FuzzTestExecutor.java

Line 140 in efbc635

private static Optional<String> translateJUnitTimeoutToLibFuzzerFlag(ExtensionContext context) {

This is quite useful for detecting application hangs (e.g. due to infinite loops) detected by Jazzer during fuzzing. For example when you normally expect a single fuzz execution to take a few milliseconds, you can make it fail fast if it unexpectedly takes more than one second.

The problem is that @Timeout is additionally still respected by JUnit and can lead to spurious timeout failures if the complete fuzzing run takes longer.

Example

Consider this example:

@Timeout(value = 2, unit = TimeUnit.SECONDS)
@FuzzTest(maxExecutions = 10)
void test(int ignored) throws Exception {
    long start = System.nanoTime();
    // Busy wait 1 second
    while ((System.nanoTime() - start) / 1_000_000_000 < 1) {
    }
}

When run in fuzzing mode each fuzzing execution takes less than the 2 seconds timeout and passes (as desired). However the complete fuzzing run takes ~10 seconds, so JUnit then (erroneously) reports the fuzzing run as failure.

Suggested solution

Maybe instead of (or in addition to?) levaraging JUnit's timeout mechanism it would be better to add a dedicated @FuzzTest parameter, e.g.:

/**
 * Timeout for a single fuzzing execution. Only has an effect in fuzzing mode, not in regression mode.
 * The timeout format is the same as for {@link #maxDuration()}, for example "1s". An empty string
 * (the default) means no timeout.
 */
String executionTimeout() default "";

A nice side effect of this might be that it is easier to discover by users than the current @Timeout support.

Possibly related to #478

[Marcono1234]

Hmm, it seems Jazzer does try to account for this problem and disables JUnit's timeout handling:

jazzer/src/main/java/com/code_intelligence/jazzer/driver/junit/JUnitRunner.java

Lines 110 to 113 in efbc635

	// JUnit's timeout handling interferes with libFuzzer as from the point of view of JUnit
	// all fuzz test invocations are combined in a single JUnit test method execution.
	// https://junit.org/junit5/docs/current/user-guide/#writing-tests-declarative-timeouts-mode
	.configurationParameter("junit.jupiter.execution.timeout.mode", timeoutMode)

But apparently this has no effect when launching the test within IntelliJ (at least for me, when launching a specific test method).
So a dedicated @FuzzTest parameter might be a more reliable solution.

On the other hand, @Timeout might be suitable because it also applies to regression mode, where you would want to test if a timeout which occurred during fuzzing mode occurs during regression testing as well (assuming the timeout was not caused by fuzzing instrumentation overhead).
Or regression mode would have to honor the proposed new @FuzzTest#executionTimeout as well.

[Marcono1234]

Going to close this; as mentioned above Jazzer seems to already solve this by disabling JUnit handling.
Maybe it would be worth though creating a follow-up issue since, as mentioned, this does not work when running the fuzzing testing with IntelliJ.

A workaround can for example be to disable JUnit's timeout like this:

@BeforeAll
static void disableJUnitTimeout() {
    System.setProperty("junit.jupiter.execution.timeout.mode", "disabled");
}

However, this then also affects Jazzer regression mode, which is undesired. So if possible this is something Jazzer would have to fix because only it knows if a test execution is running in regression or fuzzing mode.

[Marcono1234]

Maybe it would be worth though creating a follow-up issue since, as mentioned, this does not work when running the fuzzing testing with IntelliJ.

Have created #1060 now.