Describe, bail and each don't work as written in the doc

[Sharcoux]

🐛 Bug Report

Describe doesn't start a test suite, bail doesn't stop the tests after a fail, each doesn't format names correctly, and jest output is so verbose that it overflows my terminal.

To Reproduce

• git clone https://github.com/Sharcoux/GoldMines.git
• npm i
• npm run test
• Try to read the results of the tests

Expected behavior

describe

According to the documentation:

describe(name, fn) creates a block that groups together several related tests in one "test suite".

This should start 2 tests suite:

    describe('test 1',() => {
      test('one space, one mine', () => {
        expect(findBestSpot(1, 1, 1, 1, [{"x":0,"y":0}])).toEqual({"coordinates":{"x":0,"y":0},"goldMines":1});
      });
    });
  
    describe('test 2',() => {
      test('one space, no mine', () => {
        expect(findBestSpot(1, 1, 1, 1, [])).toEqual({"coordinates":{"x":0,"y":0},"goldMines":0});
      });
    });

Actual result:
=> Test Suites: 1 failed, 1 total

bail

According to the documentation:

The bail config option can be used here to have Jest stop running tests after the first failure.

Running: npx jest -b

Actual result:
=> Tests: 23 failed, 10 passed, 33 total

How could 23 tests fail if jest did stop at first failure?

each

According to the documentation:

Generate unique test titles by positionally injecting parameters with printf formatting:

So this test's name should be 'name', but instead, it's name is '1'...

const A = [[1, 2, 3, "name"]];
test.each(A)('%s', (a,b,c) => expect(a+b).toBe(c));

Actual result:

 PASS  ./index.test.js
  ✓ 1 (5ms)

By the way, "%4$s" doesn't work either....

Link to repl or repo (highly encouraged)

A small repo with a single test file that perfectly illustrate the issue

Run npx envinfo --preset jest

Paste the results here:

  System:
    OS: macOS High Sierra 10.13.6
    CPU: (4) x64 Intel(R) Core(TM) i7-5650U CPU @ 2.20GHz
  Binaries:
    Node: 8.9.2 - ~/.nvm/versions/node/v8.9.2/bin/node
    Yarn: 1.12.1 - ~/.yarn/bin/yarn
    npm: 6.3.0 - ~/.nvm/versions/node/v8.9.2/bin/npm
  npmPackages:
    jest: ^23.6.0 => 23.6.0

[SimenB]

A "test suite" in jest's terminology means a test file, not a describe. So the docs are imprecise there, should definitely be improved. In the future we wanna improve that situation, see #6616 (comment).

bail also just stops at the file level, it doesn't abort running more tests within the same file. That could potentially change, I guess. Not sure it adds value, though. Perhaps 🙂

Regarding each... I not sure what's going on there, @mattphillips could you chime in?

Regarding verbosity - we know jest is pretty verbose, and it's probably gonna stay that way. There's a couple of things you can do

1. use https://github.com/rickhanlonii/jest-silent-reporter which will only print errors, and only do that once
2. use the watch mode. When using bail it's mostly so you can focus on a few failure, right? You can use both file filtering and test name filtering in watch mode (I recommend to use https://github.com/jest-community/jest-watch-typeahead so you can see what's gonna run before starting the test run). We also have a "failures only" mode which will only rerun test files that had failures in the previous runs. Those combined should make it pretty easy to focus on a single failure at a time without being overwhelmed by lots of output

[Sharcoux]

Thanks for your reply.

A "test suite" in jest's terminology means a test file, not a describe

I'm not sure about the relevance of the describe function in this case... But ok. The doc should really be corrected, though, because it is very confusing.

bail also just stops at the file level, it doesn't abort running more tests within the same file.

Shouldn't this be reflected into the doc?

Regarding verbosity

In my use case, I provide students with a project that have already the tests written for them. In this case, if they run npm run test they will be flooded by test failures and they will only be able to read the last tests, whereas the tests are written with progressive difficulty. But the one they are interested in are probably the first ones.

Fortunately, I finally could work around this by hacking my way through each.

[Sharcoux]

Sorry guys, but I slept on this, and it really bugs me.

When you decide to use an option to stop the tests after the first failure, in what kind of context would it be the expected behavior that it keeps failing all the tests in the current file?

Moreover, you are telling me that with jest, if I want to create 20 tests and I want the tests to stop at the first failure, my only choice is to create 20 test files?!

[rickhanlonii]

Hey @Sharcoux sorry for the confusion - I think we would be interested in making bail work at a test-level but I think the way it works today is accurate to the documentation (though it could be clearer)

Do you want to submit a feature proposal to update bail to stop at the first failing test instead of the first failing test suite/test file?

[Sharcoux]

You might find me narrow-minded, but I don't think that

The bail config option can be used here to have Jest stop running tests after the first failure.

that I can read there, would normally be interpreted by:

If the bail config option is set, Jest will stop the tests if at least one test failed in the current file.

Actually, I think that most people would understand: Jest will stop running tests after the first failure 😜

I'll check if I'm able to provide a PR about this if you are willing to change the way bail is working.

[rickhanlonii]

@cpojer what do you think about --bail stopping at the first test instead of the first test file?

@Sharcoux if the docs aren't clear about this then I'm happy to merge a clarification 👌

[cpojer]

I'm fine with that. I can see advantages of both but showing just one error may be less confusing for people.

[rickhanlonii]

Yeah I agree

@Sharcoux if you want to take a stab go ahead!

[urig]

I'm not sure about the relevance of the describe function in this case... But ok. The doc should really be corrected, though, because it is very confusing.

Pull Request to resolve issue in documentation regarding the describe() method: #7452

[spinningarrow]

Just got bitten by this as well. I'd say that the current description of bail is even more misleading:

Alias: -b. Exit the test suite immediately upon n number of failing test suite. Defaults to 1.

Bail currently keeps running the remaining tests in a file even when the first one fails i.e. it does not appear to "exit the suite immediately".

Or does it mean that it should exit jest (not the test suite) immediately after a single test suite fails? Either way the language and grammar are rather confusing.

[fullstackwebdev]

Bail doesn't exit the test suite and is meaningless. This is a bug.

Bail should stop testing immediately upon n number of failing and n defaults to 1.

Please make sure everyone understands, because you won't understand that this is a bug until you understand this statement: It does not stop immediately upon n (where n=1 as default) fails.

Given you understand this, it is a bug and not a "feature"

[dartandrevinsky]

@SimenB , what, still no assignee? How many people or likes would it take to finally do anything about this bug? There are two ways: update the docs, reveal the sour truth about "the great misleading of bail", or, to change its behavior to match the docs. The latter is preferable. Who do I need to CC in order to incite the action?

[thymikee]

@dartandrevinsky Please cc yourself and send a PR with a fix. It's an open source project and you're very welcome to help fixing issues you have with it :)

[dartandrevinsky]

With all due respect (and do mean respect), Michal, the 'collaborator' badge that you and several other people sport here, suggests you are more capable to address this issue (with no need of assuming the offensive). I apologize if I sounded rude, sorry - I wanted to draw attention to this issue.

Apparently it is an issue - it was reported a year ago - there was an argument, yet the OP's reasons seem to outweigh the responses. There is no working bail, no way to abruptly stop the tests - if it's the part of the bigger design, maybe it should be shared? Or fixed accordingly?

[irfan798]

Sorry guys, but I slept on this, and it really bugs me.

When you decide to use an option to stop the tests after the first failure, in what kind of context would it be the expected behavior that it keeps failing all the tests in the current file?

Moreover, you are telling me that with jest, if I want to create 20 tests and I want the tests to stop at the first failure, my only choice is to create 20 test files?!

I still wonder if I have to write different test files for bail to work?
I am using describe.each for using the same test for different implementations of a class.
But still bail option doesn't work after a failure so i have to wait until every one of them finalizes.

Do i need to duplicate same test test to make it work parallel and work with the bail option?
What is the purpose if .each then?

Do you have any plans to

group together several related tests in one "test suite".

using something like describe

[MetaMmodern]

Opened 2 years ago and still no progress..Well, I have a similar question. How can I force the "describe" test to be counted as a suite? I don't care now if you have a bug or misunderstanding in ocumentation, I just want to tell the Jest what to consider as a test suite, cause now I have 1 test file and several "describe" tests inside and I want each of this "describe" tests to be considered as a Suite, not the file entirely. I need this to see the success/fail rate through the "describe" tests.

[irfan798]

Opened 2 years ago and still no progress..Well, I have a similar question. How can I force the "describe" test to be counted as a suite? I don't care now if you have a bug or misunderstanding in ocumentation, I just want to tell the Jest what to consider as a test suite, cause now I have 1 test file and several "describe" tests inside and I want each of this "describe" tests to be considered as a Suite, not the file entirely. I need this to see the success/fail rate through the "describe" tests.

That is the thing, you can't.
In the end I had to separate the describe blocks into different files.

[MetaMmodern]

Opened 2 years ago and still no progress..Well, I have a similar question. How can I force the "describe" test to be counted as a suite? I don't care now if you have a bug or misunderstanding in ocumentation, I just want to tell the Jest what to consider as a test suite, cause now I have 1 test file and several "describe" tests inside and I want each of this "describe" tests to be considered as a Suite, not the file entirely. I need this to see the success/fail rate through the "describe" tests.

That is the thing, you can't.
In the end I had to separate the describe blocks into different files.

That's sad( I can't seperate my describe block because they are generated inside one file in a foreach loop for cucumber feature files(((

[mekane]

I'm trying to adopt Jest for a project I'm working on and I got here because it appears "bail" still doesn't halt after a test fails. Is this just a super low priority or what? To me this is a big shortcoming for a test runner to have. I'm surprised this hasn't been fixed. Was there any progress towards a fix or a design to implement this? I'd be happy to help work on a PR if anyone else has an idea where to start.

[timatperfectco]

Every other test system I've used means I can add --bail and then the output of the failed test is the most recent thing in scroll.

Not jest.

Please, please, please, fix this?

[jeremycare]

To run our large E2E solution that needs to run in order, we needed to have this feature.

As a workaround for that, we decided to split tests into different tests files.
Then use the testSequencer to play tests in order.

We replaced:
beforeAll by globalSetup

afterAll by globalTearDown

Hope this will helps folks coming here.

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[MetaMmodern]

Not stale. Waiting for fix.