[🐛 Bug]: allure-reporter - setting a custom message in matcher options makes the test to be reported as broken instead of failed

[pfyod]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

WebdriverIO Version

latest

Node.js Version

latest

Mode

WDIO Testrunner

Which capabilities are you using?

No response

What happened?

If one sets a custom message in matcher options like so:

expect(...).someAssertion(..., {message: 'whatever'}

and this assertion fails, then allure reporter marks the test as "broken" (=uncaught exception) instead of failed (=assertion failure).

Workaround is manually adding a string "expect" to custom messages to trigger the || message.includes('expect') condition from here

What is your expected behavior?

Failed assertions should always report test as failed.

How to reproduce the bug.

// "broken" test
await expect(await $('html')).toHaveAttribute('lang', 'en', {
   message: `Page is not in English.`
});

// "failed" test, but no custom message
await expect(await $('html')).toHaveAttribute('lang', 'en');

// workaround to force "failed" test with a custom message containing the string "expect"
await expect(await $('html')).toHaveAttribute('lang', 'en', {
   message: `Expected page to be in English, but it's not.`
});

Relevant log output

N/A - log is always the same

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

[BorisOsipov]

@pfyod

Failed assertions should always report test as failed.

Sounds good. We need more details. How is the reporter supposed to guess that this stacktrace is an assert error and not a regular error?
Keep in mind that your solution should work with any assert library, even if we don't have invented it yet 😊

[pfyod]

@BorisOsipov I guess the problem is indeed hard to solve if you want to support any third party assert library (maybe some sort of extension point that allows the user to override getTestStatus?).

My point is that it should probably work in an "obvious" manner for first-party library combination of expect-webdriverio + @wdio/allure-reporter, or be somehow mentioned in docs.

[wdio-bot]

Thanks for reporting!

We greatly appreciate any contributions that help resolve the bug. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this bug being fixed. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers!

[BorisOsipov]

Probably the best solution is to port https://github.com/allure-framework/allure-js/blob/dc06dbc09c64a256a573a43dbb7cedf15f27fb2b/packages/allure-js-commons/src/sdk/utils.ts#L5 to reporter.

[pfyod]

@BorisOsipov: I think that there's actually some sort of bug (or convention that does not work that great in this scenario) somewhere. Only the first line of error message ends up inside test.error.message that gets passed to getTestStatus (the rest is inside test.error.stack only), and since custom message gets prepended to the original expect result with a newline, the actual whole message never gets inspected by getTestStatus. Would get "solved" if one reorders if (test.error.message) { and if (test.error.stack) {, but I do not know further implications of that.