@RepeatFailedTest

[nipafx]

First draft of @RepeatIfFailedTest. Code contains lots of TODOs and the test is more of a demo - it needs to be wrapped in the usual Jupiter test engine execution and assertions.

Feedback and code is welcome! 😃

---

I hereby agree to the terms of the JUnit Pioneer Contributor License Agreement.

[jbduncan]

I wonder if it would make sense to make @RetryIfFailedTest an annotation that would be applied on top of @Test, @ParameterizedTest and @TestFactory for flexibility, and thus rename @RetryIfFailedTest to @RetryIfFailed.

I imagine that for @ParameterizedTest and @TestFactory, @RetryIfFailed would retry each subtest/testlet that fails, independently from one another. So if a @TestFactory returned 3 dynamic tests, and if dynamic test 2 failed, then @RetryIfFail would retry that one and none of the others.

@nicolaiparlog, what do you think?

[smoyer64]

@jbduncan I think at some point I had proposed the generic idea of @TestDecorator - it would be interesting to see if it's possible to generalize this feature to all "test types". If so, perhaps the boilerplate could in fact be extracted to make writing other test decorators easier.

[jbduncan]

@smoyer Interesting idea! Seems to me like a long-term goal, though. So I'd suggest that someone approaches this idea as a separate PR in the future.

I'd be interested to hear if @nicolaiparlog has any other thoughts or opinions about this idea. :)

[nipafx]

@jbduncan I don't see a way to implement @RetryIfFail in the general way you describe it. Unless I'm missing something, there is no extension point to ...

• intercept and rerun failing tests
• interact with dynamic tests
• interact with parameterized tests

So the only way I found to do this is to use the test template extension point (TestTemplateInvocationContextProvider) and generate a stream of test executions that stops as soon as one execution passes (-> test passes) or the number of retries was reached (-> test fails).

[marcphilipp]

@nicolaiparlog Please be sure to tests this with parallel execution enabled. I think the Iterator will probably have to block in next() until the previous execution has finished.

[nipafx]

@marcphilipp Thanks for pointing that out - I didn't think about parallel execution at all. I also didn't yet look into that feature. Is there any way for a test or test template (besides blocking, which 🤢) to signal during parallel test execution that it doesn't want to be parallelized?

[marcphilipp]

@Execution(SAME_THREAD) should work.

[nipafx]

Turned this into a minimal viable feature. Well, almost... logging intermittent test failures is not exactly a core feature but it's already implemented. Once I find a good way to test it, I will document it.

Here are my remaining feature ideas:

• make safe for parallel execution (currently, a commend states that this shouldn't be combined)
• specify on which exceptions (not) to retry
• specify a delay (linear, exponential, ...?) between test executions
• make RepeatFailedTestInvocationContext injectable and add fields:
  • max count
  • current invocation count
  • exceptions so far
• create an "always fail" mode where tests are never repeated (e.g. by configuration via property)

Before I do that, though, I'm interested to hear from the rest, whether this looks like a worthwhile feature. @sormuras, @marcphilipp, what do you think?

[marcphilipp]

I think instead of logging intermittent failures, we should wrap them in TestAbortedExceptions so that the test is reported as aborted but not failed.

[nipafx]

Wow, I didn't know about TestAbortedExceptions! That's much better, will do!

[nipafx]

Thank you @marcphilipp, that improved the ease of use and feedback of the feature by a lot. Can you take another look?

[nipafx]

Err, GitHub, Y U No link builds?

• Travis ✔️
• AppVeyor ✔️

[smoyer64]

Is this change actually related to the new feature? (it's really hard to see what's changed on that line).

[nipafx]

The change removes a <p> tag around the first sentence, but... I'm not sure why, the site seems to be fine as it is. Nevermind, I will remove the change.

[smoyer64]

Since we're putting the examples and longer documentation in adoc files, the javadocs should probably provide a link to them.

[nipafx]

See #149.

[marcphilipp]

In the case of parallel execution this should block until the previous invocation is done. Alternatively, we could meta-annotate @RepeatFailedTest with @Execution(SAME_THREAD). However, the latter would cause all @RepeatFailedTest methods in a test class to run sequentially.

[nipafx]

I think making this extension thread-safe in a way that makes sense and interacts well with Jupiter is not trivial and should be done in a separate issue. My questions:

• should all repetitions be executed in parallel?
• if not, would a block during the traversal of the stream returned by provideTestTemplateInvocationContexts not be unexpected for Jupiter?

(While thinking about this, I wanted to write a test that executes @RepeatFailedTest in parallel. Is there a good way to do that?)

For now, I want to go with @Execution(SAME_THREAD).

[nipafx]

Oh, that requires bumping the Jupiter dependency from 5.1.1 to 5.3. I'm okay with that. Does anybody have any qualms about this?

[nipafx]

I think this is done. Once @marcphilipp made his review, I will merge. Or he can. :)

Proposed commit message:

Implement @RepeatFailedTest

It is occasionally helpful to repeat failing tests a few times before
giving up on them. The annotation `@RepeatFailedTest(n)` does just
that:

1. run the annotated method as a test
2. if it fails:
	a) if this was the n-th run, mark test as failed
	b) otherwise, mark as aborted and go to 1

PR: #110

[marcphilipp]

@nicolaiparlog Please see my review above.

[nipafx]

I'd like to bump the dependency on Jupiter from 5.1.1 to 5.3 (reason). Any nays?

[marcphilipp]

A clear "yay" from my side. 😉