[🐛 Bug]: <@wdio/allure-reporter": "^8.39.0 generates green report even for the failed tests>

[sarvesh4you]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

WebdriverIO Version

8.40.3

Node.js Version

20.12.2

Mode

WDIO Testrunner

Which capabilities are you using?

No response

What happened?

1. Description:

@wdio/allure-reporter": "^8.39.0 generates green report even for the failed tests if scrollIntoView method has been used into the beforeEach block.

2. Example code:

It always generates the incorrect pass allure report even test fails in the below expect because of scrollIntoView method has been used into the beforeEach block.

describe('My Login application', () => {

   beforeEach(async() => {
       await LoginPage.open()
       await (await $('button')).scrollIntoView({block: 'center'});
   })

   it('should login with valid credentials - pass', async () => {
       await LoginPage.login('tomsmith', 'SuperSecretPassword!')
       await expect(true).toBe(false);
   })
}) 

But if we comment out the scrollIntoView method or move it from beforeEach block into the test block then it generates the correct failed allure report as the test fails in the below expect

describe('My Login application', () => {

    beforeEach(async() => {
        await LoginPage.open()
       // await (await $('button')).scrollIntoView({block: 'center'});
    })

    it('should login with valid credentials - pass', async () => {
        await (await $('button')).scrollIntoView({block: 'center'});
        await LoginPage.login('tomsmith', 'SuperSecretPassword!')
        await expect(true).toBe(false);
    })
})

What is your expected behavior?

It should generate the failed allure report even after scrollIntoView method has been used into the beforeEach block as test will fail because of the assert await expect(true).toBe(false); in the test.

How to reproduce the bug.

Steps to reproduce:

1. Clone the repo to get the smallest code to reproduce this bug: https://github.com/sarvesh4you/wdio-allure-reporter-bug.git or use the below example:

describe('My Login application', () => {

    beforeEach(async() => {
        await LoginPage.open()
        await (await $('button')).scrollIntoView({block: 'center'});
    })

    it('should login with valid credentials - pass', async () => {
        await LoginPage.login('tomsmith', 'SuperSecretPassword!')
        await expect(true).toBe(false);
    })
})

2. Open the attached git project or paste the above code in any IDE
3. Run npm i followed by npm run test to run the test (test will always fail because of the assert await expect(true).toBe(false);)
4. Run npm run allure to generate and open allure report (it will generate green allure report even after test fails)

Relevant log output

<>\wdio_allure_bug\wdio_allure_bug
npm run test

> wdio_allure_bug@1.0.0 test
> wdio run ./wdio.conf.ts


Execution of 1 workers started at 2024-08-27T02:16:00.953Z

2024-08-27T02:16:00.976Z INFO @wdio/cli:launcher: Run onPrepare hook
2024-08-27T02:16:00.977Z INFO @wdio/utils: Setting up browser driver for: chrome@stable
2024-08-27T02:16:00.978Z INFO @wdio/utils: Setting up browser binaries for: chrome@stable
2024-08-27T02:16:00.979Z INFO webdriver: Using Chromedriver v127.0.6533.122 from cache directory C:\Users\<>\AppData\Local\Temp
2024-08-27T02:16:00.981Z INFO @wdio/cli:launcher: Run onWorkerStart hook
2024-08-27T02:16:00.982Z INFO @wdio/local-runner: Start worker 0-0 with arg: run,./wdio.conf.ts
[0-0] 2024-08-27T02:16:02.948Z INFO @wdio/local-runner: Run worker command: run
[0-0] RUNNING in chrome - file:///C:/workspace/wdio_allure_bug/wdio_allure_bug/test/specs/test.e2e.ts
[0-0] 2024-08-27T02:16:04.908Z INFO webdriver: Initiate new session using the WebDriver protocol
[0-0] 2024-08-27T02:16:05.111Z INFO webdriver: Using Chromedriver v127.0.6533.122 from cache directory C:\Users\<>\AppData\Local\Temp
[0-0] 2024-08-27T02:16:05.135Z INFO chromedriver: Starting ChromeDriver 127.0.6533.122 (79e8a05ecec6ce4de27a174466ead0e897258435-refs/branch-heads/6533@{#2002}) on port 62593
[0-0] 2024-08-27T02:16:05.135Z INFO chromedriver: Remote connections are allowed by an allowlist (0.0.0.0).
[0-0] 2024-08-27T02:16:05.136Z INFO chromedriver: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
[0-0] 2024-08-27T02:16:05.138Z INFO chromedriver: ChromeDriver was started successfully.
[0-0] 2024-08-27T02:16:06.144Z INFO @wdio/utils: Started Chromedriver v127.0.6533.122 with params --port=62593 --allowed-origins=* --allowed-ips=0.0.0.0 in 1041ms on port 62593
[0-0] 2024-08-27T02:16:06.276Z INFO webdriver: [POST] http://127.0.0.1:62593/session
[0-0] 2024-08-27T02:16:06.276Z INFO webdriver: DATA {
[0-0]   capabilities: {
[0-0]     alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': [Object] },
[0-0]     firstMatch: [ {} ]
[0-0]   },
[0-0]   desiredCapabilities: {
[0-0]     browserName: 'chrome',
[0-0]     'goog:chromeOptions': {
[0-0]       binary: 'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe'
[0-0]     }
[0-0]   }
[0-0] }

DevTools listening on ws://127.0.0.1:62604/devtools/browser/60084bc1-0cb1-440e-a38d-9c9ebc266623
[0-0] 2024-08-27T02:16:06.969Z INFO webdriver: COMMAND navigateTo("https://the-internet.herokuapp.com/login")
[0-0] 2024-08-27T02:16:06.970Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/url
[0-0] 2024-08-27T02:16:06.970Z INFO webdriver: DATA { url: 'https://the-internet.herokuapp.com/login' }
[0-0] 2024-08-27T02:16:09.286Z INFO webdriver: RESULT null
[0-0] 2024-08-27T02:16:09.290Z INFO webdriver: COMMAND findElement("css selector", "button")
[0-0] 2024-08-27T02:16:09.290Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element
[0-0] 2024-08-27T02:16:09.291Z INFO webdriver: DATA { using: 'css selector', value: 'button' }
[0-0] 2024-08-27T02:16:09.301Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': 'f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.8'
[0-0] }
[0-0] 2024-08-27T02:16:09.307Z INFO webdriver: COMMAND getElementRect("f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.8")
[0-0] 2024-08-27T02:16:09.308Z INFO webdriver: [GET] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element/f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.8/rect
[0-0] 2024-08-27T02:16:09.317Z INFO webdriver: RESULT { height: 58, width: 161, x: 33, y: 343.78125 }
[0-0] 2024-08-27T02:16:09.318Z INFO webdriver: COMMAND getWindowRect()
[0-0] 2024-08-27T02:16:09.318Z INFO webdriver: [GET] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/window/rect
[0-0] 2024-08-27T02:16:09.322Z INFO webdriver: RESULT { height: 660, width: 1050, x: 10, y: 10 }
[0-0] 2024-08-27T02:16:09.324Z INFO webdriver: COMMAND executeScript("return (() => [
[0-0]             window.scrollX, window.scrollY
[0-0]         ]).apply(null, arguments)", <object>)
[0-0] 2024-08-27T02:16:09.324Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/execute/sync
[0-0] 2024-08-27T02:16:09.324Z INFO webdriver: DATA {
[0-0]   script: 'return (() => [\n' +
[0-0]     '            window.scrollX, window.scrollY\n' +
[0-0]     '        ]).apply(null, arguments)',
[0-0]   args: []
[0-0] }
[0-0] 2024-08-27T02:16:09.329Z INFO webdriver: RESULT [ 0, 0 ]
[0-0] 2024-08-27T02:16:09.332Z INFO webdriver: COMMAND performActions(<object>)
[0-0] 2024-08-27T02:16:09.332Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/actions
[0-0] 2024-08-27T02:16:09.332Z INFO webdriver: DATA {
[0-0]   actions: [
[0-0]     { id: 'action1', type: 'wheel', parameters: {}, actions: [Array] }
[0-0]   ]
[0-0] }
[0-0] 2024-08-27T02:16:09.343Z WARN webdriverio: Failed to execute "scrollIntoView" using WebDriver Actions API: move target out of bounds
[0-0]   (Session info: chrome=127.0.6533.120)!
[0-0] Re-attempting using `Element.scrollIntoView` via Web API.
[0-0] 2024-08-27T02:16:09.344Z INFO webdriver: COMMAND executeScript("return ((elem, options) => elem.scrollIntoView(options)).apply(null, arguments)", <object>)
[0-0] 2024-08-27T02:16:09.345Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/execute/sync
[0-0] 2024-08-27T02:16:09.345Z INFO webdriver: DATA {
[0-0]   script: 'return ((elem, options) => elem.scrollIntoView(options)).apply(null, arguments)',
[0-0]   args: [
[0-0]     {
[0-0]       'element-6066-11e4-a52e-4f735466cecf': 'f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.8',
[0-0]       ELEMENT: 'f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.8'
[0-0]     },
[0-0]     { block: 'center' }
[0-0]   ]
[0-0] }
[0-0] 2024-08-27T02:16:09.350Z INFO webdriver: RESULT null
[0-0] 2024-08-27T02:16:09.355Z INFO webdriver: COMMAND findElement("css selector", "#username")
[0-0] 2024-08-27T02:16:09.355Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element
[0-0] 2024-08-27T02:16:09.356Z INFO webdriver: DATA { using: 'css selector', value: '#username' }
[0-0] 2024-08-27T02:16:09.365Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': 'f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.5'
[0-0] }
[0-0] 2024-08-27T02:16:09.368Z INFO webdriver: COMMAND elementClear("f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.5") 
[0-0] 2024-08-27T02:16:09.369Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element/f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.5/clear
[0-0] 2024-08-27T02:16:09.397Z INFO webdriver: RESULT null
[0-0] 2024-08-27T02:16:09.399Z INFO webdriver: COMMAND elementSendKeys("f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.5", "<Screenshot[base64]>")
[0-0] 2024-08-27T02:16:09.399Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element/f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.5/value
[0-0] 2024-08-27T02:16:09.399Z INFO webdriver: DATA { text: 'tomsmith' }
[0-0] 2024-08-27T02:16:09.440Z INFO webdriver: RESULT null
[0-0] 2024-08-27T02:16:09.442Z INFO webdriver: COMMAND findElement("css selector", "#password")
[0-0] 2024-08-27T02:16:09.442Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element
[0-0] 2024-08-27T02:16:09.442Z INFO webdriver: DATA { using: 'css selector', value: '#password' }
[0-0] 2024-08-27T02:16:09.453Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': 'f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.7'
[0-0] }
[0-0] 2024-08-27T02:16:09.456Z INFO webdriver: COMMAND elementClear("f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.7") 
[0-0] 2024-08-27T02:16:09.456Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element/f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.7/clear
[0-0] 2024-08-27T02:16:09.476Z INFO webdriver: RESULT null
[0-0] 2024-08-27T02:16:09.478Z INFO webdriver: COMMAND elementSendKeys("f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.7", "SuperSecretPassword!")
[0-0] 2024-08-27T02:16:09.479Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element/f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.7/value
[0-0] 2024-08-27T02:16:09.479Z INFO webdriver: DATA { text: 'SuperSecretPassword!' }
[0-0] 2024-08-27T02:16:09.516Z INFO webdriver: RESULT null
[0-0] 2024-08-27T02:16:09.518Z INFO webdriver: COMMAND findElement("css selector", "button[type="submit"]")
[0-0] 2024-08-27T02:16:09.518Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element
[0-0] 2024-08-27T02:16:09.519Z INFO webdriver: DATA { using: 'css selector', value: 'button[type="submit"]' }
[0-0] 2024-08-27T02:16:09.529Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': 'f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.8'
[0-0] }
[0-0] 2024-08-27T02:16:09.532Z INFO webdriver: COMMAND elementClick("f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.8") 
[0-0] 2024-08-27T02:16:09.532Z INFO webdriver: [POST] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d/element/f.531FEA8B6E75366DEBC678442BA9D273.d.EC089A458C48A9999D1704020009B14F.e.8/click
[0-0] 2024-08-27T02:16:09.924Z INFO webdriver: RESULT null
[0-0] Error in "My Login application.should login with valid credentials - pass"
Error: expect(received).toBe(expected) // Object.is equality

Expected: false
Received: true
    at Context.<anonymous> (C:\workspace\wdio_allure_bug\wdio_allure_bug\test\specs\test.e2e.ts:14:28)
[0-0] 2024-08-27T02:16:09.935Z INFO webdriver: COMMAND deleteSession()
[0-0] 2024-08-27T02:16:09.936Z INFO webdriver: [DELETE] http://127.0.0.1:62593/session/36b52cfebe80f599a4a701b2fe84180d
[0-0] 2024-08-27T02:16:10.046Z INFO webdriver: RESULT null
[0-0] 2024-08-27T02:16:10.046Z INFO webdriver: Kill driver process with PID 45736
[0-0] FAILED in chrome - file:///C:/workspace/wdio_allure_bug/wdio_allure_bug/test/specs/test.e2e.ts
2024-08-27T02:16:10.198Z INFO @wdio/cli:launcher: Run onWorkerEnd hook
2024-08-27T02:16:10.199Z INFO @wdio/cli:launcher: Run onComplete hook

 "spec" Reporter:
------------------------------------------------------------------
[chrome 127.0.6533.120 windows #0-0] Running: chrome (v127.0.6533.120) on windows
[chrome 127.0.6533.120 windows #0-0] Session ID: 36b52cfebe80f599a4a701b2fe84180d
[chrome 127.0.6533.120 windows #0-0]
[chrome 127.0.6533.120 windows #0-0] » \test\specs\test.e2e.ts
[chrome 127.0.6533.120 windows #0-0] My Login application
[chrome 127.0.6533.120 windows #0-0]    ✖ should login with valid credentials - pass
[chrome 127.0.6533.120 windows #0-0]
[chrome 127.0.6533.120 windows #0-0] 1 failing (3s)
[chrome 127.0.6533.120 windows #0-0]
[chrome 127.0.6533.120 windows #0-0] 1) My Login application should login with valid credentials - pass
[chrome 127.0.6533.120 windows #0-0] expect(received).toBe(expected) // Object.is equality

Expected: false
Received: true
[chrome 127.0.6533.120 windows #0-0] Error: expect(received).toBe(expected) // Object.is equality
[chrome 127.0.6533.120 windows #0-0]
[chrome 127.0.6533.120 windows #0-0] Expected: false
[chrome 127.0.6533.120 windows #0-0] Received: true
[chrome 127.0.6533.120 windows #0-0]     at Context.<anonymous> (C:\workspace\wdio_allure_bug\wdio_allure_bug\test\specs\test.e2e.ts:14:28)      


Spec Files:      0 passed, 1 failed, 1 total (100% completed) in 00:00:09

2024-08-27T02:16:10.200Z INFO @wdio/local-runner: Shutting down spawned worker
2024-08-27T02:16:10.463Z INFO @wdio/local-runner: Waiting for 0 to shut down gracefully
2024-08-27T02:16:10.464Z INFO @wdio/local-runner: shutting down


C:\<>\wdio_allure_bug\wdio_allure_bug
npm run allure

> wdio_allure_bug@1.0.0 allure
> allure generate allure-results --clean -o allure-report && allure open

Report successfully generated to allure-report
Starting web server...
2024-08-27 05:16:17.446:INFO::main: Logging initialized @393ms to org.eclipse.jetty.util.log.StdErrLog
Server started at <http://192.168.8.111:62650/>. Press <Ctrl+C> to exit

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

[wdio-bot]

Thanks for reporting!

We greatly appreciate any contributions that help resolve the bug. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this bug being fixed. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers!

[christian-bromann]

Fixed in #13993