Bail config not working with Jasmine in v5

[mrahman2327]

Describe your question with as much detail as possible
I recently upgraded wdio from v3 to v5. All is working great except for the bail option. Previously in v3, it worked with setting bail: 1 and stopOnSpecFailure: true in jasmineNodeOpts. However, it is not working as such in v5.

Q: Did something change in the functionality here?

Follow up Q: I would also like to use the functionality of both bail and specFileRetries in order to retry the test up to 3 times on the first failure. Will that be achievable? As I remember reading something where specFileRetries and bail setup can conflict with one another.

If it's about a specific piece of code, try and include some of it to support your question.
[...]

Environment (please complete the following information):

• WebdriverIO version: ^5.0.0 [e.g. 4.13.2]
• Mode: WDIO Testrunner [Standalone mode or WDIO Testrunner]
• If WDIO Testrunner, running sync/async: sync [e.g. sync/async]
• Node.js version: 10.15.3 [e.g. 8.11.2]
• NPM version: 6.4.1 [e.g. 5.8.0]
• Browser name and version: Chrome 74 [e.g. Chrome 68]
• Platform name and version: Mac OS X and Linux [e.g. Windows 10]
• Additional wdio packages used (if applicable): @wdio/jasmine-framework, @wdio/spec reporter, @wdio/local-runner, @wdio/cli, @wdio/sync

Config of WebdriverIO

// ====================
// Runner Configuration
// ====================
//
// WebdriverIO allows it to run your tests in arbitrary locations (e.g. locally or
// on a remote machine).
runner: 'local',
//
// ==================
// Specify Test Files
// ==================
//
suites: suites,
// Specs to include/exclude.
// specs: ,
// exclude: ,
//
// ============
// Capabilities
// ============
//
maxInstances: 10,
//
capabilities: [
    {
        maxInstances: 5,
        //
        browserName: 'chrome',
        'goog:chromeOptions': {
            // to run chrome headless the following flags are required
            // (see https://developers.google.com/web/updates/2017/04/headless-chrome)
            // args: ['--headless', '--disable-gpu'],
        }
        // If outputDir is provided WebdriverIO can capture driver session logs
        // it is possible to configure which logTypes to include/exclude.
        // excludeDriverLogs: ['*'], // pass '*' to exclude all driver session logs
        // excludeDriverLogs: ['bugreport', 'server'],
    }
],
//
// ===================
// Test Configurations
// ===================
// Define all options that are relevant for the WebdriverIO instance here
//
// Level of logging verbosity: trace | debug | info | warn | error | silent
logLevel: 'trace',
//
// Set directory to store all logs into
outputDir: `${path.output}`,
// If you only want to run your tests until a specific amount of tests have failed use
// bail (default is 0 - don't bail, run all tests).
bail: 1,
//
baseUrl: url.base,
//
// Default timeout for all waitFor* commands.
waitforTimeout: waitTimeout,
//
// Default timeout in milliseconds for request
// if Selenium Grid doesn't send response
connectionRetryTimeout: 90000,
//
// Default request retries count
connectionRetryCount: 3,
//
// Make sure you have the wdio adapter package for the specific framework installed
// before running any tests.
framework: 'jasmine',
//
// The number of times to retry the entire specfile when it fails as a whole
specFileRetries: 0,
//
reporters: ['spec'],
//
// Options to be passed to Jasmine.
jasmineNodeOpts: {
    //
    // Jasmine default timeout
    defaultTimeoutInterval: parameter.testTimeout,
    //
    stopOnSpecFailure: true,
    // The Jasmine framework allows interception of each assertion in order to log the state of the application
    // or website depending on the result. For example, it is pretty handy to take a screenshot every time
    // an assertion fails.
    expectationResultHandler: function(passed, assertion) {
        /**
         * only take screenshot if assertion failed
         */
        if (passed) {
            return;
        }
    }
},

Additional context
Context on follow up question: As some spec files are big, it would be great if I could use bail along to save time. Bail option is not working both with and without specFileRetries.

[omerose]

I'm also experiencing the same issue upgrading from wdio v4->v5

I have tried permutations of these options:

        bail: 1,
        stopSpecOnExpectationFailure: true,
        stopOnSpecFailure: true,

With a single test, it is retried ~ 8 times (95s) where the first failure happens around 20s. This can save us a lot of time when debugging our tests.

[mrahman2327]

For people still facing this issue out there, my workaround was to set failFast: true in JasmineNodeOpts, in addition to bail: 1. It now stops execution on first failure and retries as well with specFileRetries.

[christian-bromann]

As some spec files are big, it would be great if I could use bail along to save time. Bail option is not working both with and without specFileRetries.

Ah, I understand the issue now. The WebdriverIO bail option is only looking at wdio spec files. As stated in the documentation:

If you only want to run your tests until a specific amount of tests have failed use bail (default is 0 - don't bail, run all tests). Note: Please be aware that when using a third party test runner such as Mocha, additional configuration might be required.

If you want to bail within spec files you need to use framework specific options.