Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Protractor test failures are not retried and failed retries clear results from previous failure #8103

Closed
dianabarsan opened this issue Feb 25, 2023 · 2 comments
Assignees
Labels
Type: Technical issue Improve something that users won't notice
Milestone

Comments

@dianabarsan
Copy link
Member

Describe the issue
We added retries to protractor e2e test failures.
However, they do not seem to work: the retry will fail immediately with an error:

 Re-running tests , attempt :  1
 Re-running the following test files : /home/runner/work/cht-core/cht-core/tests/e2e/protractor/client-side-transitions/forms/index.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/client-side-transitions/muting.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/contacts/contact-summary.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/contacts/reports-subject.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/africas-talking.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/rapidpro.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/registration-by-sms.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/send-message.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/targets/target-aggregates.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/translations/incorrect-locale.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/client-side-transitions/forms/index.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/client-side-transitions/muting.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/contacts/contact-summary.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/contacts/reports-subject.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/africas-talking.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/rapidpro.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/registration-by-sms.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/send-message.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/targets/target-aggregates.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/translations/incorrect-locale.spec.js


Closing report
[12:10:57] E/launcher - callback is not a function
[12:10:57] E/launcher - TypeError: callback is not a function
    at /home/runner/work/cht-core/cht-core/node_modules/protractor-jasmine2-screenshot-reporter/index.js:525:11
    at FSReqCallback.oncomplete (node:fs:192:23)

Example run: https://github.com/medic/cht-core/actions/runs/4269504105/jobs/7433297666#step:16:634
The report folder is also overwritten, and the other failure screenshots are deleted, which makes debugging harder.

Describe the improvement you'd like
Fix it so retries will happen.

Describe alternatives you've considered
Have it so the retrying will not clear previous results folder. even if retries don't work.

@dianabarsan dianabarsan added the Type: Technical issue Improve something that users won't notice label Feb 25, 2023
@dianabarsan dianabarsan changed the title Protractor test failures are not retried and clear results from previous failure Protractor test failures are not retried and failed retries clear results from previous failure Feb 25, 2023
@garethbowen garethbowen self-assigned this Feb 26, 2023
garethbowen added a commit that referenced this issue Feb 26, 2023
@garethbowen
Copy link
Member

Example of successful capture of run failure: https://github.com/medic/cht-core/actions/runs/4277828531/jobs/7447091600#step:16:2097

I'm now getting the screenshot and logs say:

 Executed 1 of 2 specs (1 FAILED) (1 PENDING) in 1 sec.
>> 01:28:07.816 INFO [ActiveSessions$1.onStop] - Removing session 52785e2b69b9f77df80e35649e74f1b8 (org.openqa.selenium.chrome.ChromeDriverService)
[01:28:07] I/launcher - 0 instance(s) of WebDriver still running
[01:28:07] I/launcher - chrome #01 failed 1 test(s)
[01:28:07] I/launcher - overall: 1 failed spec(s)
Closing report
[ProtractorJasmineRetry] Test failed after 5 attempts, exiting with code 1

@garethbowen garethbowen added this to the 4.2.0 milestone Feb 27, 2023
garethbowen added a commit that referenced this issue Mar 1, 2023
Protractor retries never helped test success rates, and ended up obscuring results
making debugging harder.
@garethbowen
Copy link
Member

Merged. No AT required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Technical issue Improve something that users won't notice
Projects
Status: Done
Development

No branches or pull requests

2 participants