Testcafe xUnit reporter marks tests as both failure and error #7629
Comments
anthonyferroHU
added
the
TYPE: bug
need-response-app
bot
added
the
STATE: Need response
|
Hello @anthonyferroHU , You are right, testcafe treats We need to discuss what we can do about this issue with the team. We will update this thread once we make a decision. For the team: it looks like this issue has the same cause. |
need-response-app
bot
removed
the
STATE: Need response
|
Hello @anthonyferroHU, After additional research we came to the conclusion that this behavior depends on the CI system you use, so as we mentioned above, we don't plan to change it. You can use the onBeforeWrite hook to remove the "testsuite" tag attribute: // .testcaferc.js
const TEST_SUIT_REGEX = new RegExp(/<testsuite (name=".*") (tests=".*") (failures=".*") (skipped=".*") (errors=".*") (time=".*") (timestamp=".*") >/);
function xUnitOnBeforeWrite (writeInfo) {
if (writeInfo.initiator !== 'reportTaskDone')
return;
if (!TEST_SUIT_REGEX.test(writeInfo.formattedText))
return;
writeInfo.formattedText = writeInfo.formattedText.replace(TEST_SUIT_REGEX, (...args) => {
const [
_,
name,
tests,
failures,
skipped,
errors,
time,
timestamp,
] = args;
return `<testsuite ${name} ${tests} ${failures} ${skipped} ${time} ${timestamp} >`;
});
}
module.exports = {
hooks: {
reporter: {
onBeforeWrite: {
'xunit': xUnitOnBeforeWrite,
},
},
},
};Feel free to reopen this issue if this approach is not suitable. |
What is your Scenario?
When using the testcafe xUnit reporter, if a test fails, the xml report logs 1 failure and 1 error count for the test. Similarly, if a test is stopped from an error (not a failing assertion), the xml report logs 1 failure and 1 error count for the test.
In this case, all 6 of these either failed or were stopped from an error, resulting in the following counts in the xml results:
<testsuite name="TestCafe Tests: Chrome 112.0.0.0 / macOS 10.15.7" tests="6" failures="6" skipped="0" errors="6"We are moving to run these tests via AWS Codebuild, and these counts result in an error generating the report in Codebuild:
Error in UPLOAD_ARTIFACTS phase: [rspec_reports: [error processing report: [InvalidInputException: Test summary: status count cannot be negative]]]This is happening because the reporter is marking each failing test as both an error and a failure; and combining this in the total count results in the number of failures + errors being larger than the number of tests. I didn't get much help from AWS support because they are expecting either a failure or an error per test. This feels like a valid request on the Testcafe side to ensure accurate xml results wherever we're running our tests.
What is the Current behavior?
Any test that does not pass is logged as both a failure and an error
What is the Expected behavior?
A test that does not pass should be logged as either a failure or an error, not both
What is your public website URL? (or attach your complete example)
n/a - Run a group of tests that fail using the testcafe xUnit reporter and specify an output file, check the total test, failure and error counts in the file after tests run
What is your TestCafe test code?
n/a - not an issue with test code, just the xUnit report summary
Your complete configuration file
No response
Your complete test report
Screenshots
No response
Steps to Reproduce
Run a group of failing tests w/ this reporter configuration, check counts in
spec_report.xml:"reporter": [
{
"name": "xunit",
"output": "spec_report.xml"
}
],
TestCafe version
1.15.0
Node.js version
14, 18
Command-line arguments
-r xunit:spec_report.xml
Browser name(s) and version(s)
No response
Platform(s) and version(s)
No response
Other
Similar issues described here aquasecurity/trivy#880 and here https://stackoverflow.com/questions/64332952/why-does-my-aws-codebuild-test-report-result-in-an-incomplete-status
The text was updated successfully, but these errors were encountered: