Jasmine afterTest() wdio hook returns wrong passed value for failed spec as passed=true

[venkatrvaradhan]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

WebdriverIO Version

8.20.2

Node.js Version

8.15.0

Mode

WDIO Testrunner

Which capabilities are you using?

browserName: chrome,
browserVersion: 'latest',
platformName: 'Windows 10'
operatingSystem: Windows 10

dependencies in package.json

"@babel/core": "^7.23.2",
"@babel/preset-env": "^7.23.2",
"@babel/register": "^7.22.15",
"@wdio/allure-reporter": "^8.20.0",
"@wdio/cli": "^8.20.2",
"@wdio/jasmine-framework": "^8.23.0",
"@wdio/local-runner": "^8.20.0",
"@wdio/spec-reporter": "^8.20.0",
"allure-commandline": "^2.24.1"

What happened?

Hi Team, We are using the WebdriverIO with the Jasmine framework for our test automation project. We observed the afterTest() hook in the WebdriverIO shows the passed value as true for the spec that failed with an expectation. Due to this issue, we cannot take a screenshot of the failed spec and attach it to the allure report. Please find the below framework details.

WebdriverIO version: 8.20.2
Framework: Jasmine
Language: Javascript
Mode: [WDIO Testrunner]
If WDIO Testrunner, running sync/async: [async]
Node.js version: [18.15.0]
NPM version: [9.5.0]
Browser name and version: [Chrome 119]
Platform name and version: [Windows 10

We even tried to use the below check in afterTest() hook, but still, the screenshot is not captured in the allure report failed spec.

afterTest: async function (test, context, { error, result, duration, passed, retries }) {
if (!passed || error) {
await browser.takeScreenshot()
}
}
Attached is the allure report screenshot

What is your expected behavior?

The afterTest() hook should return the correct passed value as true/false for the spec passed/failed due to expectation/assertion

How to reproduce the bug.

Attached is the sample project created to reproduce this issue. The project contains two specs, one will pass and the other will fail due to expectations.

JasmineProject.zip

Relevant log output

Execution of 1 workers started at 2023-11-19T14:42:21.622Z

[0-0] RUNNING in chrome - file:///C:/Users/User1/Desktop/JasmineProject/test/specs/test.e2e.js

DevTools listening on ws://127.0.0.1:56785/devtools/browser/8fcda011-3409-4c9b-afb6-7b7d4dff81c6
[26656:30380:1119/201242.995:ERROR:policy_logger.cc(157)] :components\policy\core\browser\policy_conversions_client.cc(317) DeveloperToolsDisabled has an error of type: Ignored because it was overridden by DeveloperToolsAvailability.
[0-0] Error in "My Login application should login with valid credentials and expectation failure"
    at <Jasmine>
    at UserContext.<anonymous> (C:\Users\User1\Desktop\JasmineProject\test\specs\/test.e2e.js:24:50)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
[0-0] FAILED in chrome - file:///C:/Users/User1/Desktop/JasmineProject/test/specs/test.e2e.js

 "spec" Reporter:
------------------------------------------------------------------
[chrome 119.0.6045.160 windows #0-0] Running: chrome (v119.0.6045.160) on windows
[chrome 119.0.6045.160 windows #0-0] Session ID: 6dc13b4a034ab8e2c836fc5a6d043a9f
[chrome 119.0.6045.160 windows #0-0] » \test\specs\/test.e2e.js
[chrome 119.0.6045.160 windows #0-0] My Login application
[chrome 119.0.6045.160 windows #0-0]    ✓ should login with valid credentials
[chrome 119.0.6045.160 windows #0-0]    ✖ should login with valid credentials and expectation failure
[chrome 119.0.6045.160 windows #0-0]
[chrome 119.0.6045.160 windows #0-0] 1 passing (37.8s)
[chrome 119.0.6045.160 windows #0-0] 1 failing
[chrome 119.0.6045.160 windows #0-0]
[chrome 119.0.6045.160 windows #0-0] 1) My Login application should login with valid credentials and expectation failure
[chrome 119.0.6045.160 windows #0-0] Expect $(`//a/i[text()=' Logout']`) to be disabled

Expected: "disabled"
Received: "not disabled"
[chrome 119.0.6045.160 windows #0-0]     at <Jasmine>
[chrome 119.0.6045.160 windows #0-0]     at UserContext.<anonymous> (C:\Users\User1\Desktop\JasmineProject\test\specs\/test.e2e.js:24:50)
[chrome 119.0.6045.160 windows #0-0]     at processTicksAndRejections (node:internal/process/task_queues:95:5)


Spec Files:      0 passed, 1 failed, 1 total (100% completed) in 00:00:55

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

[BorisOsipov]

Thanks for reporting!

We greatly appreciate any contributions that help resolve the bug. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this bug being fixed. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers!

[tech-dm-klymenko]

Hi @venkatrvaradhan,
To get a screenshot, you can use such approach as workaround:

// => wdio.conf.js

jasmineOpts: {
    expectationResultHandler: function(passed, assertion) {
        /**
         * only take screenshot if assertion failed
         */
        if(passed) {
            return
        }
        browser.saveScreenshot(`assertionError_${assertion.error.message}.png`)
    }
},

And if possible, a little bit later I will try to look further into the reason for improper passing of a value to the afterTest() function.

[tech-dm-klymenko]

I will take this issue