Stop testcase execution on failure in beforeEach

[FrankyBoy]

I think it would be very useful to stop execution of a testcase if the beforeEach fails (i.e. throws an exception or similar).

Example: precondition is used for login -> if the login does not succeed, it does not make any sense to run the actual test.

I already tried to expect() something in the beforeEach (or in a function that it calls, to be more exact), and throwing an exception. The first did not work at all for my purposes (as other steps in the beforeEach were still executed), the 2nd at least registers as an error but the execution still continues.

Or maybe something like an option to stop executing the testcase as soon as one expectation fails could work, but I didn't find any such option.

[otomi]

Having a manual way of stopping a test case execution would be really handy. For example when creating tests for Selenium Webdriver and during the flow for example login fails thus I cannot perform rest of my steps I could fail the test with predefined message and skip the rest of the execution and jump to next one.

[infews]

Is a manual fail method sufficient? If so, we have this in our backlog.

[slackersoft]

I don't think the manual fail method solves this by itself. It sounds to me like the desired behavior is that a single spec stops execution upon encountering its first expectation failure or exception and proceeds straight to the next spec. This would probably need to be some other kind of option to jasmine similar to whats requested in #414 but for a single spec instead of the whole suite.

I'm not sure what we would want to do as far as afterEach calls in this mode as well.

[slackersoft]

I've created a set of stories in tracker that detail how we want to address this issue.

Story for core: https://www.pivotaltracker.com/story/show/1165916
Story to update npm: https://www.pivotaltracker.com/story/show/89323102
Story to update gem: https://www.pivotaltracker.com/story/show/89323040
Story to update py: https://www.pivotaltracker.com/story/show/89323226

We'd be happy to review pull requests for this functionality.

Thanks for using jasmine!

[just-boris]

+1. Now this is weird behavior. If I have a failed setup in beforeEach, all my tests would be incorrect. Why do you execute this test cases anyway?

[donaldpipowitch]

Sorry, do I understand this correctly: if I call done.fail in beforeEach the remaining its are executed anyway?

[xealot]

Just ran into this, the behavior is a bit outside of expectation IMO. We have a beforeEach that calls done.fail because of a busted setup but continues to attempt to run all the tests within the describe.

[myitcv]

Adding to the voices here...

fail("Some message");
expect(false).toBe(true);

Definitely does not make sense to continue execution after the fail above, whether in a test or a beforeEach

But in the interests of making this a backwards compatible change (i.e. no surprises for people who may be relying on this behaviour), would a new function fail_hard("Some messages") do the trick?

[slackersoft]

@myitcv this feature isn't just about when you call fail, but also when an expectation fails. We've done a little work to get closer to this desired functionality, but we're not quite there yet. We would still be happy to review a pull request that makes this happen.

[myitcv]

@slackersoft - thanks for the response. Sounds like you've considered the case of calling fail too which is great.

[DmitryEfimenko]

+1. Any progress on this?
By the way, the same should go for beforeAll()

[LukeSkyw]

This is a real issue. Just spent half a day trying to understand why my tests were failing because angular was throwing an exception on an unknown module loaded in a beforeEach but nothing was shown in the logs. Had to instrument angular with console.log to find out.
This issue was fixed in mocha: mochajs/mocha#270

[slackersoft]

@LukeSkyw I think you might be conflating something like #529 with this issue. If your exception makes it up to Jasmine, your spec will be marked as failed with that error. However, Jasmine continues with the rest of the execution queue of beforeEaches, its, and afterEaches; this is more what this issue is about.

[filmaj]

+1, this would be handy

[skyshore2001]

@slackersoft
I tried version 2.5.2 to test my back-end web APIs. When fail() / pending() is called in beforeEach, the "it" block still execute, although the result is marked failure / pending.

[skyshore2001]

Actually my requirement is to skip the other cases/specs in the same suite if some critical case fails.
This issue may also match my requirement: #414

I'm studying the code to see if there is some light-weight approach.
Any suggestion is welcome.

[slackersoft]

@skyshore2001 Jasmine definitely does not want to have specific specs/suite marked as "critical". We would be happy to review a pull request for #414 to stop suite execution at the end of the first failing spec.

Skipping out in the middle of the spec run (this issue) is a bit more complicated, because depending on the type of error, Jasmine probably still needs to run any afterEach (or afterAll depending) to cleanup state for the next spec. This would then require the QueueRunner to know which of the functions it is given are setup and teardown and not just have a list of functions to call.

[monkpit]

@slackersoft couldn't QueueRunner be made to process a queue of queues, and if it's executing the "test spec" queue and there is a failure, we bail on the "test spec" queue and move on to the afterEach/afterAll queue?

[slackersoft]

This feature was released in version 2.7 of Jasmine-Core. If you don't have it, please update your dependency.

Thanks for using Jasmine!

[monkpit]

For future googlers that might land on this issue - see stopSpecOnExpectationFailure:

https://jasmine.github.io/2.8/node.html#section-13

[korobochka]

Despite the issue being closed, I want to note that I am still seeing initial wrong behaviour with "jasmine": "^3.4.0".

Code:

describe("Feature", function () {
    beforeEach(function () {
        console.log("beforeEach");
        throw new Error("");
    });

   it("Test 1", function () {
       console.log("Test 1");
   });

    it("Test 2", function () {
        console.log("Test 2");
    });
});

Console output:

Randomized with seed 13617
Started
beforeEach
Test 2
FbeforeEach
Test 1
F

[sgravrock]

@korobochka I was only able to reproduce your output with stopSpecOnExpectationFailure set to false in the Jasmine configuration. When I set it to true, I get the following:

Randomized with seed 29833
Started
beforeEach
FbeforeEach
F

[korobochka]

@sgravrock Yes, you are right. Sorry for confusion.