[🐛 Bug]: preferScenarioName option for Browserstack service isn't working

[denise-maia-ribeiro]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

WebdriverIO Version

latest

Node.js Version

14

Mode

WDIO Testrunner

Which capabilities are you using?

No response

What happened?

I'm using browserstack service with preferScenarioName: true option to show the scenario name as the session name in Browserstack, but the session name displayed in Browserstack is the name of the feature instead. I'm algo using wdio-cucumber-parallel-execution to run the scenarios in parallel that splits the feature file in 1 feature file per scenario.

Here is a piece of config file:

import * as dotenv from 'dotenv';
dotenv.config({
    path: ".env.browserstack"
})
import {decrypt} from '../src/utils/CryptUtils';
import { BROWSERSTACK_CUSTOM_ID, BUILD_ALIAS } from '../src/utils/DefaultConstants';
import { HOVERFLY_IOS, REMOTE_CONFIG_FILE } from '../src/utils/HoverflyConstants';
import * as allure from 'allure-commandline';
import * as browserstack from "browserstack-local";
import * as wdioParallel from 'wdio-cucumber-parallel-execution';
import * as argv from 'yargs';
import * as fs from 'file-system';
import * as sync from 'fs-extra';
import {startHccSimulationByFile} from '../src/utils/HoverflyUtils';

var bs_local = new browserstack.Local();
var bs_args = { 'key': decrypt(process.env.BROWSERSTACK_ACCESS_KEY), 'verbose': true, 'force': true, 'parallelRuns': '3', 'localIdentifier': process.env.CUCUMBER_TAGS.concat(process.env.PLATFORM).concat("-").concat(process.env.S3_APP_MAJOR_VERSION) }
const sourceSpecDirectory = `features/**`;
const parallelExecutionReportDirectory = `reports/json/`;
let featureFilePath = `${sourceSpecDirectory}/*.feature`;
if (argv.argv.parallel === 'true') {
    const tmpSpecDirectory = `tmp`;
    wdioParallel.performSetup({
        sourceSpecDirectory: sourceSpecDirectory,
        tmpSpecDirectory: tmpSpecDirectory,
        cleanTmpSpecDirectory: true
    });
    featureFilePath = `${tmpSpecDirectory}/*.feature`
}
exports.config = {
    user: process.env.BROWSERSTACK_USERNAME,
    key: decrypt(process.env.BROWSERSTACK_ACCESS_KEY),
    autoCompileOpts: {
        autoCompile: true,
        tsNodeOpts: {
            transpileOnly: true,
            project: 'tsconfig.json'
        }
    },
    specs: [
        featureFilePath
    ],
    maxInstances: 6,
    capabilities: [
        {
            platformName: "iOS",
            "appium:app": process.env.BS_HASH_ID || BROWSERSTACK_CUSTOM_ID,
            "appium:platformVersion": process.env.OS_VERSION_IOS,
            "appium:deviceName" : process.env.DEVICE_NAME_IOS,
            "appium:autoDismissAlerts": true,
            "appium:autoGrantPermissions":true,
            "appium:noReset": false,
            "appium:newCommandTimeout": 180000,
            "appium:processArguments": HOVERFLY_IOS,
            'bstack:options' : { 
                "buildName": process.env.BUILD_ALIAS || BUILD_ALIAS,
                "local" : true, 
                "appiumVersion": process.env.BS_APPIUM_VERSION,
                "gpsLocation": process.env.BS_GPS_LOCATION,
                "networkLogs": process.env.BS_NETWORK_LOGS,
                "userName" : process.env.BROWSERSTACK_USERNAME,
                "accessKey" : decrypt(process.env.BROWSERSTACK_ACCESS_KEY)
            }
        }
    ],
    services: [
        [
        'browserstack', {
            browserstackLocal: true,
            preferScenarioName: true
    }
        ]
    ],
    framework: 'cucumber',
    specFileRetries: 1,
    reporters: [
        ['allure', {
        outputDir: 'reports/allure-results',
        disableWebdriverStepsReporting: true,
        disableWebdriverScreenshotsReporting: false,
        useCucumberStepReporter: true,
        }],
        ['cucumberjs-json',{
            jsonFolder: parallelExecutionReportDirectory,
        }]
    ],
    cucumberOpts: {
        // <string[]> (file/dir) require files before executing features
        require: ['./src/step-definitions/*.ts', './src/hooks/**.ts'],
        // <boolean> show full backtrace for errors
        backtrace: true,
        // <boolean< Treat ambiguous definitions as errors
        failAmbiguousDefinitions: true,  
        // <string[]> ("extension:module") require files with the given EXTENSION after requiring MODULE (repeatable)
        requireModule: [],
        // <boolean> invoke formatters without executing steps
        dryRun: false,
        // <boolean> abort the run on first failure
        failFast: false,
        // <string[]> (type[:path]) specify the output format, optionally supply PATH to redirect formatter output (repeatable)
        format: ['pretty'],
        // <boolean> hide step definition snippets for pending steps
        snippets: true,
        // <boolean> hide source uris
        source: false,
        // <string[]> (name) specify the profile to use
        profile: [],
        // <string[]> ("extension:module") require files with the given EXTENSION after requiring MODULE (repeatable)
        name: [], 
        // <boolean> fail if there are any undefined or pending steps
        strict: true,
        // <string> (expression) only execute the features or scenarios with tags matching the expression
        tagExpression: process.env.CUCUMBER_TAGS || '@e2e',
        // <number> timeout for step definitions
        timeout: 90000,
        // <boolean> Enable this config to treat undefined definitions as warnings.
        ignoreUndefinedDefinitions: false,
         // <boolean> add cucumber tags to feature or scenario name
        tagsInTitle: false           

    }

What is your expected behavior?

As described in Webdriverio documentation, when I use preferScenarioName: true in browserstack service, the scenario's name should be displayed as session name in Browserstack.

How to reproduce the bug.

import * as dotenv from 'dotenv';
dotenv.config({
    path: ".env.browserstack"
})
import {decrypt} from '../src/utils/CryptUtils';
import { BROWSERSTACK_CUSTOM_ID, BUILD_ALIAS } from '../src/utils/DefaultConstants';
import { HOVERFLY_IOS, REMOTE_CONFIG_FILE } from '../src/utils/HoverflyConstants';
import * as allure from 'allure-commandline';
import * as browserstack from "browserstack-local";
import * as wdioParallel from 'wdio-cucumber-parallel-execution';
import * as argv from 'yargs';
import * as fs from 'file-system';
import * as sync from 'fs-extra';
import {startHccSimulationByFile} from '../src/utils/HoverflyUtils';

var bs_local = new browserstack.Local();
var bs_args = { 'key': decrypt(process.env.BROWSERSTACK_ACCESS_KEY), 'verbose': true, 'force': true, 'parallelRuns': '3', 'localIdentifier': process.env.CUCUMBER_TAGS.concat(process.env.PLATFORM).concat("-").concat(process.env.S3_APP_MAJOR_VERSION) }
const sourceSpecDirectory = `features/**`;
const parallelExecutionReportDirectory = `reports/json/`;
let featureFilePath = `${sourceSpecDirectory}/*.feature`;
if (argv.argv.parallel === 'true') {
    const tmpSpecDirectory = `tmp`;
    wdioParallel.performSetup({
        sourceSpecDirectory: sourceSpecDirectory,
        tmpSpecDirectory: tmpSpecDirectory,
        cleanTmpSpecDirectory: true
    });
    featureFilePath = `${tmpSpecDirectory}/*.feature`
}
exports.config = {
    user: process.env.BROWSERSTACK_USERNAME,
    key: decrypt(process.env.BROWSERSTACK_ACCESS_KEY),
    autoCompileOpts: {
        autoCompile: true,
        tsNodeOpts: {
            transpileOnly: true,
            project: 'tsconfig.json'
        }
    },
    specs: [
        featureFilePath
    ],
    maxInstances: 6,
    capabilities: [
        {
            platformName: "iOS",
            "appium:app": process.env.BS_HASH_ID || BROWSERSTACK_CUSTOM_ID,
            "appium:platformVersion": process.env.OS_VERSION_IOS,
            "appium:deviceName" : process.env.DEVICE_NAME_IOS,
            "appium:autoDismissAlerts": true,
            "appium:autoGrantPermissions":true,
            "appium:noReset": false,
            "appium:newCommandTimeout": 180000,
            "appium:processArguments": HOVERFLY_IOS,
            'bstack:options' : { 
                "buildName": process.env.BUILD_ALIAS || BUILD_ALIAS,
                "local" : true, 
                "appiumVersion": process.env.BS_APPIUM_VERSION,
                "gpsLocation": process.env.BS_GPS_LOCATION,
                "networkLogs": process.env.BS_NETWORK_LOGS,
                "userName" : process.env.BROWSERSTACK_USERNAME,
                "accessKey" : decrypt(process.env.BROWSERSTACK_ACCESS_KEY)
            }
        }
    ],
    services: [
        [
        'browserstack', {
            browserstackLocal: true,
            preferScenarioName: true
    }
        ]
    ],
    framework: 'cucumber',
    specFileRetries: 1,
    reporters: [
        ['allure', {
        outputDir: 'reports/allure-results',
        disableWebdriverStepsReporting: true,
        disableWebdriverScreenshotsReporting: false,
        useCucumberStepReporter: true,
        }],
        ['cucumberjs-json',{
            jsonFolder: parallelExecutionReportDirectory,
        }]
    ],
    cucumberOpts: {
        // <string[]> (file/dir) require files before executing features
        require: ['./src/step-definitions/*.ts', './src/hooks/**.ts'],
        // <boolean> show full backtrace for errors
        backtrace: true,
        // <boolean< Treat ambiguous definitions as errors
        failAmbiguousDefinitions: true,  
        // <string[]> ("extension:module") require files with the given EXTENSION after requiring MODULE (repeatable)
        requireModule: [],
        // <boolean> invoke formatters without executing steps
        dryRun: false,
        // <boolean> abort the run on first failure
        failFast: false,
        // <string[]> (type[:path]) specify the output format, optionally supply PATH to redirect formatter output (repeatable)
        format: ['pretty'],
        // <boolean> hide step definition snippets for pending steps
        snippets: true,
        // <boolean> hide source uris
        source: false,
        // <string[]> (name) specify the profile to use
        profile: [],
        // <string[]> ("extension:module") require files with the given EXTENSION after requiring MODULE (repeatable)
        name: [], 
        // <boolean> fail if there are any undefined or pending steps
        strict: true,
        // <string> (expression) only execute the features or scenarios with tags matching the expression
        tagExpression: process.env.CUCUMBER_TAGS || '@e2e',
        // <number> timeout for step definitions
        timeout: 90000,
        // <boolean> Enable this config to treat undefined definitions as warnings.
        ignoreUndefinedDefinitions: false,
         // <boolean> add cucumber tags to feature or scenario name
        tagsInTitle: false           

    }

Relevant log output

N/A

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

[christian-bromann]

Thanks for reporting!

Not entirely sure if we changed something recently in the BS service, it is possible thought that this could be a regression.

Any contributions that resolves the bug are highly appreciated. Please take a look into our contribution guidelines and let us know if you have any questions. Cheers!

[denise-maia-ribeiro]

Hi @christian-bromann , I checked in the code and it seems that the variable this._scenariosThatRan used in the if that changes the fullTitle to the scenario's name is populated only if the scenario's status is 'skipped':

 after (result: number) {
        // For Cucumber: Checks scenarios that ran (i.e. not skipped) on the session
        // Only 1 Scenario ran and option enabled => Redefine session name to Scenario's name
        if (this._options.preferScenarioName && this._scenariosThatRan.length === 1){
            this._fullTitle = this._scenariosThatRan.pop()
        }
(...)
afterScenario (world: Frameworks.World) {
        const status = world.result?.status.toLowerCase()
        if (status === 'skipped') {
            this._scenariosThatRan.push(world.pickle.name || 'unknown pickle name')
        }
(...)

[christian-bromann]

@denise-maia-ribeiro thanks for looking into this, can you make a PR with adjustments?

[denise-maia-ribeiro]

@christian-bromann I just opened the PR: #7740. If you could review it, please!

[nextlevelbeard]

Fixed by #7740