--fail-fast keeps running other test files

[novemberborn]

The test process that encounters the error simply exits if --fail-fast is enabled. Since each file runs in its own process, no other tests from that file are reported after the first failure is encountered.

However, other test processes run to completion. I'm not sure whether to:

• terminate them outright, which prevents any cleanup, even if there are no failures in those processes
• silence their output, which at least means the test output isn't cluttered with passing tests, but makes it harder to see why ava won't exit
• shrug and say that this is expected

When running with limited concurrency we'll also run into this scenario, so we should implement the same behavior. Additionally we mustn't launch new test processes.

Any thoughts on what to do with processes that are already running?

[novemberborn]

You can reproduce this issue by creating one.test.js:

import test from 'ava'

test(t => t.fail())

and two.test.js:

import test from 'ava'

test(t => t.pass())

Then ava --verbose --fail-fast.

[ThomasBem]

I also noticed this while working on issue #1134 today. Here is a screenshot of me trying to run two different files both with a failing test somewhere in the middle.

When things are as small as these examples it does not seem like much of a problem. But if you are running a huge test suite for a complex application. Then allowing Ava to keep running tests kind of defeats the purpose of --fail-fast in my opinion.

So to keep the --fail-fast option working as intended, i would say shutdown any running processes. But i don´t know what problems that might cause since you talk about preventing cleanup:

terminate them outright, which prevents any cleanup, even if there are no failures in those processes

[mightyiam]

My thoughts:

1. do not run any more test files
2. do not run any more tests
3. allow tests to complete, including their afterEach
4. allow test files to complete, including their after

Only then, exit.

[novemberborn]

So to keep the --fail-fast option working as intended, i would say shutdown any running processes. But i don´t know what problems that might cause since you talk about preventing cleanup

Test files with failing tests are terminated quickly (but not immediately, in a synchronous, blocking sense). So --fail-fast already implies that any cleanup code you have in your test file may not run if there's an error. I'm fine with extending this behavior to other test files, too.

I'm leaning towards forcefully stopping all other test processes and not logging any further output once a test has failed.

[mightyiam]

Test files with failing tests are terminated quickly (but not immediately, in a synchronous, blocking sense). So --fail-fast already implies that any cleanup code you have in your test file may not run if there's an error. I'm fine with extending this behavior to other test files, too.

I'm leaning towards forcefully stopping all other test processes and not logging any further output once a test has failed.

How is this reasonable? This should perhaps be a feature, if you find it useful, but not --fail-fast. This should be --fail-very-fast or some other naming scheme.

There should be a feature to fail on the first failing test, yet, let the run finish gracefully, like I described. And it should be the more approachable, defaulty, recommended option.

Is how I see it.

[novemberborn]

That seems unnecessarily complicated. The way I see it you'd use --fail-fast to quickly find errors so you can fix them. It helps if any state created by your tests is preserved (by virtue of not being cleaned up).

[mightyiam]

It will probably not help that state created by your tests is preserved because that would be state created by tests other than the one which actually failed. Confusing, I think.

[novemberborn]

The state of the failing test would be preserved amongst potentially that of others.

[mightyiam]

The state of the failing test is the only one we'd be interested in. The others' could be confusing and a hassle to manually clean up. That's why I propose: let them finish and clean up after themselves. No new tests will start. Just the ones already running.

[novemberborn]

Ah, so finish the currently running, non-failing tests, including their afterEach callbacks. That way most state would be cleared up, though you may see more than one failure in the test output.

We wouldn't be able to run the after callbacks though — they may assume the failing tests have completed and therefore crash.

[novemberborn]

#1693 ensures --fail-fast works with limited concurrency mode, preventing new test files from being run. It also fails fast after a timeout or if a test file crashed.

As part of #1684 I'll make sure we stop reporting subsequent test results once a test fails, while still letting the currently active tests run to completion.

[novemberborn]

As part of #1684 I'll make sure we stop reporting subsequent test results once a test fails, while still letting the currently active tests run to completion.

OK so what I ended up doing is that workers won't run other serial tests, or (as long as the error occurs early enough) concurrent tests. Tests that have started are run to completion, including hooks, and shown in the test output. If a test fails in one worker AVA will try and interrupt other workers.