failOnOrphans breaks flaky tests with maven failsafe plugin

[fredrikkarlsson-pervanovo]

It seems that the ERROR: Found orphan snapshots creates a failure that is not handled very well by failsafe (and possibly surefire) with -Dfailsafe.rerunFailingTestsCount=2.

The following command:
CI=true mvn -DskipSurefire=true -Dit.test=FlakyIT -Dfailsafe.rerunFailingTestsCount=2 verify

Creates the following output:

[INFO] --- maven-failsafe-plugin:3.2.2:integration-test (default) @ bjelin-service ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running se.pervanovo.bjelin.bjelinservice.FlakyIT
2025-11-03T18:17:31.542+01:00  INFO   --- [Pool-1-worker-1] au.com.origin.snapshots.SnapshotFile     : Snapshot File: src/test/java\se\pervanovo\bjelin\bjelinservice\__snapshots__\FlakyIT.snap
2025-11-03T18:17:31.677+01:00 ERROR   --- [Pool-1-worker-1] a.com.origin.snapshots.SnapshotVerifier  : All unused Snapshots:
se.pervanovo.bjelin.bjelinservice.FlakyIT.flaky=[
  "test"
]

Have you deleted tests? Have you renamed a test method?
[ERROR] Tests run: 3, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.181 s <<< FAILURE! -- in se.pervanovo.bjelin.bjelinservice.FlakyIT
[ERROR] se.pervanovo.bjelin.bjelinservice.FlakyIT.flaky -- Time elapsed: 0.072 s <<< FAILURE!
org.opentest4j.AssertionFailedError
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:34)
        at org.junit.jupiter.api.Assertions.fail(Assertions.java:119)
        at se.pervanovo.bjelin.bjelinservice.FlakyIT.flaky(FlakyIT.java:20)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)

[ERROR] se.pervanovo.bjelin.bjelinservice.FlakyIT -- Time elapsed: 0.181 s <<< FAILURE!
au.com.origin.snapshots.exceptions.SnapshotMatchException: ERROR: Found orphan snapshots
        at au.com.origin.snapshots.SnapshotVerifier.validateSnapshots(SnapshotVerifier.java:139)
        at au.com.origin.snapshots.junit5.SnapshotExtension.afterAll(SnapshotExtension.java:40)
        at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)

[INFO] Running se.pervanovo.bjelin.bjelinservice.FlakyIT
2025-11-03T18:17:31.721+01:00  INFO   --- [Pool-2-worker-1] au.com.origin.snapshots.SnapshotFile     : Snapshot File: src/test/java\se\pervanovo\bjelin\bjelinservice\__snapshots__\FlakyIT.snap
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.449 s -- in se.pervanovo.bjelin.bjelinservice.FlakyIT
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   FlakyIT ERROR: Found orphan snapshots
[WARNING] Flakes: 
[WARNING] se.pervanovo.bjelin.bjelinservice.FlakyIT.flaky
[ERROR]   Run 1: FlakyIT.flaky:20
[INFO]   Run 2: PASS
[INFO] 
[INFO] 
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Flakes: 1
[INFO] 
[INFO] --- maven-failsafe-plugin:3.2.2:verify (default) @ bjelin-service ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  30.456 s
[INFO] Finished at: 2025-11-03T18:17:34+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.2.2:verify (default) on project bjelin-service: There are test failures.

The test class looks like this:

package se.pervanovo.bjelin.bjelinservice;

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;

import au.com.origin.snapshots.Expect;
import au.com.origin.snapshots.junit5.SnapshotExtension;

@ExtendWith(SnapshotExtension.class)
public class FlakyIT {
    private static int counter = 0;
    Expect expect;

    @Test
    void flaky() {
        if (counter++ < 1) {
            Assertions.fail();
        }
        expect.toMatchSnapshot("test");
    }

    @Test
    void works() {
        expect.toMatchSnapshot("works");
    }
}

If the "working" test is removed, the failOnOrphans is set to false (since we cannot determine if the test was run single or just has one test) and then the flaky integration test passes without the [ERROR] FlakyIT ERROR: Found orphan snapshots line.

Any idea what to do? A reasonable workaround for me would be to disable "failOnOrphans" but that does not seem to be possible by configuration.