Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add soft assertions #373

Merged
merged 2 commits into from
Jul 2, 2018
Merged

Add soft assertions #373

merged 2 commits into from
Jul 2, 2018

Conversation

ajalt
Copy link
Contributor

@ajalt ajalt commented Jul 1, 2018

This PR adds the ability to perform multiple assertions in a block without the test stopping on the first failed assertion. The verifyAll function takes a lambda, and any assertion failures inside the block will be gathered, then thrown when the block exits.

Many other testing framework have some version of this functionality:

The implementation works by changing the should* dsl functions to send their failures to an ErrorCollector, which will gather the errors when inside a verifyAll block, or throw the error immediately otherwise. The collected errors are held in thread-local storage to prevent race conditions in the unlikely case that multiple threads invoke verifyAll concurrently.

Existing assertion functions that take a lambda often rely on throwing an exception for control flow, and so will continue to throw immediately instead of collecting their errors. This includes eventually, shouldThrow*, shouldTimeout, and property and table testing.

This PR adds the ability to perform multiple assertions in a block without the test stopping on the first failed assertion. The `verifyAll` function takes a lambda, and any assertion failures inside the block will be gathered, then thrown when the block exits.

Many other testing framework have some version of this functionality:

* [AssertJ](https://joel-costigliola.github.io/assertj/assertj-core-features-highlight.html#soft-assertions)
* [TestNG](https://jitpack.io/com/github/cbeust/testng/master/javadoc/org/testng/asserts/SoftAssert.html)
* [Spock](http://spockframework.org/spock/docs/1.1/all_in_one.html#_what_s_new_in_this_release_4)

The implementation works by changing the `should*` dsl functions to send their failures to an `ErrorCollector`, which will gather the errors when inside a `verifyAll` block, or throw the error immediately otherwise. The collected errors are held in thread-local storage to prevent race conditions in the unlikely case that multiple threads invoke `verifyAll` concurrently.

Existing assertion functions that take a lambda often rely on throwing an exception for control flow, and so will continue to throw immediately instead of collecting their errors. This includes `eventually`, `shouldThrow*`, `shouldTimeout`, and property and table testing.
@ajalt
Copy link
Contributor Author

ajalt commented Jul 1, 2018

verifyAll is not the most descriptive name. Let me know if you think of a better one.

@sksamuel
Copy link
Member

sksamuel commented Jul 2, 2018

verifyAll is currently used by property testing as well. I'll try to think of a different name.

@ajalt
Copy link
Contributor Author

ajalt commented Jul 2, 2018

assertSoftly, assertBlock, assertGroup?

@sksamuel
Copy link
Member

sksamuel commented Jul 2, 2018

soft, as daft as the name is, is what assertJ uses so why not stick with that. So I'm happy with softAssert { }

@sksamuel sksamuel merged commit a9269d4 into kotest:master Jul 2, 2018
@sksamuel
Copy link
Member

sksamuel commented Jul 2, 2018

👍

@ajalt ajalt deleted the soft-assertions branch July 2, 2018 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants