🚀 Feature: Exit code - differentiate between being unable to run tests and failing the tests

[aDu]

I want to differentiate between the tests successfully running (but failing assertions) - giving an exit code of NumOfFailedTests, verses other errors due to the test suite (i.e. out of memory, no permissions to access the test suite file, undefined variable).

Alternatively, is there an option to force an exit code of 0 instead of an exit code of NumOfFailedTests, that way when I see that the exit code is 0, I know the tests ran properly (regardless if it passed or failed).

Is there a way to do this?

Thank you.

[ScottFreeCode]

There's issue #2392 for suppressing the return code for certain consuming systems to look at the output instead.

Short of that, depending on whether you're running Mocha through the CLI or the programmatic interface, there may be a couple different options:

• Mocha's programmatic interface would report test failures through the events (e.g. particularly to the reporter, which could be a custom listener), but more critical errors would typically just be thrown as far as I know, thus allowing you to differentiate between them trivially.
• Looking at reporter output, while it involves a smidge of parsing, it should be possible to tell if the reporter ended in the usual list of test pass/pending/fail counts and errors from failed tests vs. if Mocha doesn't finish the report with that standard format because of a catastrophic error printing a standard Node stacktrace and exit instead.

Note that either of these should work for detecting (and treating as different from test failure) problems like your examples of the test file failing to load/parse or Mocha running out of memory, but does not differentiate assertion failures from regular errors that occur inside the test function run (e.g. a TypeError thrown by incorrect usage of the code that's being tested), as Mocha doesn't try very hard to detect whether an error thrown by a test run is actually from an assertion -- it just assumes both that assertions will communicate failure by throwing errors and that other mistakes/errors/bugs detected in the test should be reported as failures too.

[amedee]

Unfortunately #2392 is closed...

[JoshuaKGoldberg]

#3559 is a superset of this. Marking this one as a duplicate of #3559. Cheers!