Document test exception handling

[Marcono1234]

It seems there is currently not one single place which describes how exceptions in tests are handled. This can lead to confusion regarding behavior for unexpected exceptions, especially when they are checked exceptions, see for example #3508.
This information is currently split across the documentation, such as the assertDoesNotThrow method documentation, which says:

Although any exception thrown from a test method will cause the test to fail

JUnit 4 has a separate FAQ entry for this¹.

It might therefore be good to have one place (probably a new User Guide section?) where the exception handling is documented:

• Uncaught exceptions cause a test error by default
  • for checked exceptions a throws clause has to be added to the test method
  • usage of throws clause has no effect (e.g. it does not indicate an "expected exception")
  • (maybe) recommend using throws Exception instead of throws CheckedExceptionA, CheckedExceptionB, CheckedExceptionC since the latter introduces a lot of unnecessary clutter²
  • can use assertDoesNotThrow to explicitly indicate unexpected exceptions, but it is not required
  • (maybe) mention special exception types which cause test to be skipped or aborted
• Expected exceptions can be tested for using assertThrows / assertThrowsExactly
• Difference between test failure (i.e. assertion failure) and test error (i.e. unexpected exception)³

But this section should probably be targeted at regular users, so it should not go into detail regarding how extensions can handle exceptions. That seems to already be covered in the User Guide section about extensions.

What do you think?

Deliverables

• A section in the User Guide about test exception handling

Footnotes

1. Though that entry is slightly confusing because it recommends using throws ... for IndexOutOfBoundsException, which is an unchecked exception. ↩

2. It seems not everyone agrees that using throws Exception there is fine. But personally I am not sure if developers really deliberately list all checked exceptions, or if most often they just use the IDE action for adding a throws clause. ↩

3. I am not sure though how strong this separation still exists for JUnit 5. I thought I head read about it in the JUnit 5 documentation before, but I cannot find it right now. And in some cases there seems to be no such differentiation, e.g. assertAll()'s exception-handling behavior turns all test programming errors into test failures #3436. ↩

[marcphilipp]

Team decision:

• Document that uncaught exceptions cause test failures by default
• Document that assertions are implemented using exceptions
• Document that JUnit 5 does not differentiate between failed assertions and other exceptions but IDEs and other tools may choose to do so by checking for instances of AssertionError
• Add assertDoesNotThrow example for Java

[driptaroop]

If no-one has taken this up yet I would like to work on it if that's ok. Although since I have never contributed here a gentle guidance would be helpful.
As far as I have understood, the new documentation section for exceptions testing should be put in here right?

[marcphilipp]

Yes, I think we should add an "Other Exceptions" section after "Assertions" and "Assumptions".

@driptaroop Feel free to take this on if you're still interested. Sorry for the delay!

[driptaroop]

@marcphilipp thanks. I will take it up and assign the issue to myself.

PS: couldn't figure out how to assign the issue to myself but I will take it up nonetheless :)

[sbrannen]

This has been addressed in f915d67 and 27f3b7c.

[sbrannen]

The new "Exception Handling" section can now be viewed in the snapshot documentation:

https://junit.org/junit5/docs/snapshot/user-guide/#writing-tests-exceptions