AVA does not Bail after Assertion Failure with FailFast Option Enabled

[PeterMerkert]

• What you're trying to do
  Stopping AVA within a test when an assertion fails.

• What happened
  AVA does run on, despite an assertion being failed.

• What you expected to happen
  AVA should stop running further in the test, when an assertion failed when the setting FailFast is enabled.

A minimal code sample is:

const test = require('ava')

test('AVA does not stop when assertion fails', async t => {
	t.is(42, 43)
	console.log(42)
})

This will print 42 on my console, despite the first test actually failing. I do expect here, that the log output line is not executed any more.

My package.json includes:

"ava": {
    "failFast": true
  },

I assume the FailFast option is more about concurrent test runners and stopping them gracefully just after a test has failed. I seem to be unable to locate an option to actually force AVA to bail out of a single test once an assertion fails. I do not want to stop any other test executions and also not any other tests in the same file.

Is there such an option to stop the execution immediately for the single test once an assertion within the test failed?

[novemberborn]

Is there such an option to stop the execution immediately for the single test once an assertion within the test failed?

No. Assertions don't throw errors, so they don't cause the test to stop executing.

There are some ways of doing this but whether they make sense depend on your use case. Could you elaborate?

(I'm closing this issue for housekeeping purposes, but let's keep the conversation going.)

[PeterMerkert]

Thanks for helping out.

The short explanation: I would like to avoid an if-cascade to always double check. It feels like I do not need AVA assertions than any more (except of neat highlighting). So, I want to avoid:

let res = apiCall1
t.is(res.status, 200)
if (res.status !== 200) return
res = apiCall2
t.is(res.status, 200)
if (res.status !== 200) return
...

And anyways, AVA only highlights for me the first assertion failing in a test and not the subsequent ones. So, if there are two assertions failing within a single test, only the first one is shown in my output log.

The long explanation: I want to archive this is because I am running some integration tests, creating users not only in my API, but also in Auth0 as a SaaS provider for user handling. Within that test, I cross-check if emails are send out properly and if the user is created in all the correct fashions that the login works (username + password, magic email code, sms code).

In the end of the test, I want to remove the user from Auth0, as otherwise it gets cluttered with test users (though I already use a test tenant). And once a user is created with a phone number, it can not be created again. Thus, I clean up.

However, if something fails in the user creation step in the beginning (like the API returns 409), and then I do a check t.is(response.statusCode, 200) this assertion fails. However, the test execution just goes on. This will lead to longer test runtime than necessary and obviously probably a lot of subsequent failing other calls, which will also clutter any log collected during testing.

[novemberborn]

AVA only highlights for me the first assertion failing in a test and not the subsequent ones. So, if there are two assertions failing within a single test, only the first one is shown in my output log.

We'd like to show all failures though. See #261.

The long explanation

Right, so you're using AVA to orchestrate a more complex integration test. Which is great! But not quite what it's designed for.

Mind you, I use AVA to run integration tests, with carefully ordered serial tests, and that works great. But I'm mocking the APIs I call out to.

We have an experimental t.try() assertion you could use to maybe help with this orchestration. Unfortunately we didn't quite get the documentation done yet. There's some hints here: https://github.com/avajs/ava/releases/tag/v2.4.0.

[cakoose]

I have a data structure that efficiently maintains a set of ranges. To test it, I instantiate a simpler/dumber implementation along with the real implementation, perform a bunch of random operations to both, and then check that they end up the same.

When a test fails, I usually add additional logging to figure out what led to the failure.

With Ava's current behavior, it's way too hard to figure out what's going on because the logging doesn't stop at the first failure. This ended up being a big enough issue that I switched to Tape for now.

[novemberborn]

@cakoose sounds like a use case t.try() is perfect for. If you want to discuss further let's move that conversation to https://github.com/avajs/ava/discussions.