chore(wdio-allure-reporter): in case of onTestRetry first has status unknown in allure report

[maksym-alavatskyi]

Proposed changes

On wdio+mocha+allure tools set, in case of mocha config has retry >=1 allure report shows incorrect info on failed tests
allure-report test stats json files:
Before fix

first run:
{ "uuid": "c8dff4fd-4909-4faf-b547-9da020213022", "historyId": "a7c8b10231a2fdd69c9dfd8f13529c5a", "statusDetails": {}, "stage": "pending", "steps": [ { "status": "failed", "statusDetails": {}, "stage": "finished", "steps": [], "attachments": [], "parameters": [], "start": 1705657167422, "name": "test", "stop": 1705657167444 } ], "attachments": [ { "name": "Execution video", "type": "video/webm", "source": "57dbaa4f-dda9-4bc9-a1f3-5acf263a31e1-attachment.webm" }, { "name": "test_log", "type": "text/plain", "source": "ab3419d3-c98a-4857-8662-de962f08d5e2-attachment.txt" }, { "name": "test_error_log", "type": "text/plain", "source": "f6f689a7-16e8-4935-8cc0-a485e8364567-attachment.txt" }, { "name": "test_wdio_log", "type": "text/plain", "source": "9e3ed7e2-2dad-44e8-a230-2f63ded04d01-attachment.txt" }, { "name": "test_browser_log", "type": "text/plain", "source": "ec489314-1108-4272-a567-95ad4d7c7b48-attachment.txt" } ], "parameters": [ { "name": "browser", "value": "chrome-headless-shell-120.0.6099.109" }, { "name": "Run session id", "value": "undefined" }, { "name": "debug", "value": "false" }, { "name": "baseUrl", "value": "\"http://127.0.0.1:8080\"" }, { "name": "folder", "value": "\"./dist-e2e\"" }, { "name": "port", "value": "8080" }, { "name": "chromeVersion", "value": "\"stable\"" }, { "name": "automationProtocol", "value": "\"webdriver\"" }, { "name": "environment", "value": "\"local\"" }, { "name": "verbose", "value": "false" }, { "name": "injectToken", "value": "false" }, { "name": "testIds", "value": "[]" }, { "name": "priorities", "value": "[]" }, { "name": "maxBrowsers", "value": "3" }, { "name": "browserVersion", "value": "120.0.6099.109" }, { "name": "browserVersion", "value": "120.0.6099.109" } ], "labels": [ { "name": "language", "value": "javascript" }, { "name": "framework", "value": "wdio" }, { "name": "package", "value": ".Users.maksym.alavatskyi.Documents.app-frontends.packages.automation-core-sandbox.e2e.tests.action-list.spec.ts" }, { "name": "thread", "value": "0-0" }, { "name": "suite", "value": "[automation-core-sandbox] ActionList" }, { "name": "feature", "value": "[automation-core-sandbox] ActionList" }, { "name": "feature", "value": "automation-core-sandbox" } ], "links": [], "start": 1705657167322, "name": "[1]-Action List ", "testCaseId": "a7c8b10231a2fdd69c9dfd8f13529c5a", "stop": 1705657171757 }
retry run:
{ "uuid": "46cef57b-4b32-4061-a811-97e4949bb139", "historyId": "fa93f794a337a5d695f4edc91c38215e", "status": "broken", "statusDetails": { "message": "Custom error", "trace": "Error: Custom error\n at ActionListSpec.testCsm (/Users/maksym.alavatskyi/Documents/app-frontends/packages/automation-core-sandbox/e2e/tests/action-list.spec.ts:17:11)\n at testFn (/Users/maksym.alavatskyi/Documents/app-frontends/packages/wdio-manager/src/spec/decorators/spec.ts:119:48)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)" }, "stage": "finished", "steps": [ { "status": "failed", "statusDetails": {}, "stage": "finished", "steps": [], "attachments": [], "parameters": [], "start": 1705657167994, "name": "test", "stop": 1705657167995 } ], "attachments": [ { "name": "Execution video", "type": "video/webm", "source": "07eb3abd-4ee0-4c29-8964-e449c6eec105-attachment.webm" }, { "name": "test_log", "type": "text/plain", "source": "021b4f27-4fb1-4ddc-a521-a07ee41589d5-attachment.txt" }, { "name": "test_error_log", "type": "text/plain", "source": "d5fb9c49-c5a2-48af-ad8b-3a1e6188fcc5-attachment.txt" }, { "name": "test_wdio_log", "type": "text/plain", "source": "bb4850e9-9c7d-4d7a-bccb-0d8078f41160-attachment.txt" }, { "name": "test_browser_log", "type": "text/plain", "source": "95f0013e-8131-4464-b5cf-f09810046f0d-attachment.txt" } ], "parameters": [ { "name": "browser", "value": "chrome-headless-shell-120.0.6099.109" }, { "name": "Run session id", "value": "undefined" }, { "name": "debug", "value": "false" }, { "name": "baseUrl", "value": "\"http://127.0.0.1:8080\"" }, { "name": "folder", "value": "\"./dist-e2e\"" }, { "name": "port", "value": "8080" }, { "name": "chromeVersion", "value": "\"stable\"" }, { "name": "automationProtocol", "value": "\"webdriver\"" }, { "name": "environment", "value": "\"local\"" }, { "name": "verbose", "value": "false" }, { "name": "injectToken", "value": "false" }, { "name": "testIds", "value": "[]" }, { "name": "priorities", "value": "[]" }, { "name": "maxBrowsers", "value": "3" } ], "labels": [ { "name": "language", "value": "javascript" }, { "name": "framework", "value": "wdio" }, { "name": "package", "value": ".Users.maksym.alavatskyi.Documents.app-frontends.packages.automation-core-sandbox.e2e.tests.action-list.spec.ts" }, { "name": "thread", "value": "0-0" }, { "name": "suite", "value": "[automation-core-sandbox] ActionList" }, { "name": "feature", "value": "[automation-core-sandbox] ActionList" }, { "name": "feature", "value": "automation-core-sandbox" } ], "links": [], "start": 1705657167937, "name": "[1]-Action List ", "testCaseId": "fa93f794a337a5d695f4edc91c38215e", "stop": 1705657168101 }
After fix

first run
{ "uuid": "2068fe45-9fb5-473c-9126-e1cd98c94d38", "historyId": "fa93f794a337a5d695f4edc91c38215e", "status": "broken", "statusDetails": { "message": "Custom error", "trace": "Error: Custom error\n at ActionListSpec.testCsm (/Users/maksym.alavatskyi/Documents/app-frontends/packages/automation-core-sandbox/e2e/tests/action-list.spec.ts:17:11)\n at testFn (/Users/maksym.alavatskyi/Documents/app-frontends/packages/wdio-manager/src/spec/decorators/spec.ts:119:48)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)" }, "stage": "finished", "steps": [ { "status": "failed", "statusDetails": {}, "stage": "finished", "steps": [], "attachments": [], "parameters": [], "start": 1705656730012, "name": "test", "stop": 1705656730013 } ], "attachments": [ { "name": "Execution video", "type": "video/webm", "source": "ba9aa959-75c4-4f6b-a1ae-e3d25842db90-attachment.webm" }, { "name": "test_log", "type": "text/plain", "source": "1679b686-8127-4b9d-b2ed-b8658c97e475-attachment.txt" }, { "name": "test_error_log", "type": "text/plain", "source": "7dc0e65e-ff63-4227-9e76-6ba9323fea50-attachment.txt" }, { "name": "test_wdio_log", "type": "text/plain", "source": "c0ee2429-bb30-413d-8255-2ac78197dc10-attachment.txt" }, { "name": "test_browser_log", "type": "text/plain", "source": "61c7572d-0267-40ac-99d1-0738b3e4ef31-attachment.txt" } ], "parameters": [ { "name": "browser", "value": "chrome-headless-shell-120.0.6099.109" }, { "name": "Run session id", "value": "undefined" }, { "name": "debug", "value": "false" }, { "name": "baseUrl", "value": "\"http://127.0.0.1:8080\"" }, { "name": "folder", "value": "\"./dist-e2e\"" }, { "name": "port", "value": "8080" }, { "name": "chromeVersion", "value": "\"stable\"" }, { "name": "automationProtocol", "value": "\"webdriver\"" }, { "name": "environment", "value": "\"local\"" }, { "name": "verbose", "value": "false" }, { "name": "injectToken", "value": "false" }, { "name": "testIds", "value": "[]" }, { "name": "priorities", "value": "[]" }, { "name": "maxBrowsers", "value": "3" } ], "labels": [ { "name": "language", "value": "javascript" }, { "name": "framework", "value": "wdio" }, { "name": "package", "value": ".Users.maksym.alavatskyi.Documents.app-frontends.packages.automation-core-sandbox.e2e.tests.action-list.spec.ts" }, { "name": "thread", "value": "0-0" }, { "name": "suite", "value": "[automation-core-sandbox] ActionList" }, { "name": "feature", "value": "[automation-core-sandbox] ActionList" }, { "name": "feature", "value": "automation-core-sandbox" } ], "links": [], "start": 1705656729998, "name": "[1]-Action List ", "testCaseId": "fa93f794a337a5d695f4edc91c38215e", "stop": 1705656730056 }
retry
{ "uuid": "66d8f690-da27-4c66-ad37-d93b670f0034", "historyId": "fa93f794a337a5d695f4edc91c38215e", "status": "broken", "statusDetails": { "message": "Custom error", "trace": "Error: Custom error\n at ActionListSpec.testCsm (/Users/maksym.alavatskyi/Documents/app-frontends/packages/automation-core-sandbox/e2e/tests/action-list.spec.ts:17:11)\n at testFn (/Users/maksym.alavatskyi/Documents/app-frontends/packages/wdio-manager/src/spec/decorators/spec.ts:119:48)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)" }, "stage": "finished", "steps": [ { "status": "failed", "statusDetails": {}, "stage": "finished", "steps": [], "attachments": [], "parameters": [], "start": 1705656729859, "name": "test", "stop": 1705656729874 } ], "attachments": [ { "name": "Execution video", "type": "video/webm", "source": "7f9569ea-f7a3-4b26-883f-1f639dacb063-attachment.webm" }, { "name": "test_log", "type": "text/plain", "source": "548984bd-5602-4147-8be6-b4d1d4283cac-attachment.txt" }, { "name": "test_error_log", "type": "text/plain", "source": "e881703a-7c55-4e55-80a7-53c6a2cf2a21-attachment.txt" }, { "name": "test_wdio_log", "type": "text/plain", "source": "b02c2a27-6add-4de8-8c42-ae9660a9b9b7-attachment.txt" }, { "name": "test_browser_log", "type": "text/plain", "source": "032383d0-d9ad-4b60-bc65-3ce87383a2b0-attachment.txt" } ], "parameters": [ { "name": "browser", "value": "chrome-headless-shell-120.0.6099.109" }, { "name": "Run session id", "value": "undefined" }, { "name": "debug", "value": "false" }, { "name": "baseUrl", "value": "\"http://127.0.0.1:8080\"" }, { "name": "folder", "value": "\"./dist-e2e\"" }, { "name": "port", "value": "8080" }, { "name": "chromeVersion", "value": "\"stable\"" }, { "name": "automationProtocol", "value": "\"webdriver\"" }, { "name": "environment", "value": "\"local\"" }, { "name": "verbose", "value": "false" }, { "name": "injectToken", "value": "false" }, { "name": "testIds", "value": "[]" }, { "name": "priorities", "value": "[]" }, { "name": "maxBrowsers", "value": "3" } ], "labels": [ { "name": "language", "value": "javascript" }, { "name": "framework", "value": "wdio" }, { "name": "package", "value": ".Users.maksym.alavatskyi.Documents.app-frontends.packages.automation-core-sandbox.e2e.tests.action-list.spec.ts" }, { "name": "thread", "value": "0-0" }, { "name": "suite", "value": "[automation-core-sandbox] ActionList" }, { "name": "feature", "value": "[automation-core-sandbox] ActionList" }, { "name": "feature", "value": "automation-core-sandbox" } ], "links": [], "start": 1705656729814, "name": "[1]- Action List ", "testCaseId": "fa93f794a337a5d695f4edc91c38215e", "stop": 1705656729967 }

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• I have added proper type definitions for new commands (if appropriate)

Further comments

Reviewers: @webdriverio/project-committers

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: maksym-alavatskyi / name: maksym-alavatskyi (a5166fd, d6f34b0, 25d3376)
• ✅ login: christian-bromann / name: Christian Bromann (62f1b15)

[christian-bromann]

@maksym-alavatskyi thanks for the PR. Do you think we can add a special note to the report that the test was retried?

[maksym-alavatskyi]

@maksym-alavatskyi thanks for the PR. Do you think we can add a special note to the report that the test was retried?

If you think we need it and direct me on how do you want to see it...
In general, we have test Retries tab, that contains previous tries info and, in case test result changed, we also have specific UI mark with popup text: Test status was changed after retry

[christian-bromann]

Unfortunately I don't know much about the Allure reporter, let's see what @BorisOsipov advices here 👀

[BorisOsipov]

@maksym-alavatskyi The changes look good to me. I will be happy to merge these changes after you add the unit test.

[BorisOsipov]

Add unit tests.

[maksym-alavatskyi]

@BorisOsipov, test added, could you please review?