[V2] When a test fail, a retry from gitlab didn't stop the job

[bastienmoulia]

Summary

Sorry cypress don't failed when there is no new tests to run and there is one that already failed

How to reproduce

1. Start the Job with E2E tests from the Gitlab pipeline (1 is failing)
2. At the end of the tests, i can see on Cypress dashboard that 1 test is failing and the Gitlab Job failed
3. On the Gitlab pipeline I Retry the job containing the test
4. At the end of the tests, i can see on Cypress dashboard that 1 test is failing and the Gitlab Job passed

Relevant logs / screenshots

Logs of the Retry:

Environment

• sorry-cypress version: latest
• cypress version: 8.6
• platform: docker
• service: all

[TomaszG]

It's because the director service considers this spec as already executed, so it doesn't send it to any runner with the same ci-build-id. Otherwise, other workers/test runners would constantly retry the failed tests.

The workaround for this is to reset the failed spec file from the dashboard and then retry the job - the failed spec is then considered ready to run and will be picked up by the retried job.

I'm not sure how it could work out-of-box. Director would have to know that the retried job wants to pick up only failed jobs. However, I'm not aware of any way to differentiate a standard GitLab job from a retried one. I think that Cypress dashboard is struggling with the same issue, especially if you look at this question https://forum.gitlab.com/t/get-unique-environment-variable-to-detect-a-retry-of-job/32520 asked by Cypress.io maintainer.

I'd love to see any automated process to retry the failed tests when the GitLab job is retried but I doubt that such exists.

[agoldis]

Check out the last note on https://currents.dev/readme/concepts/cypress-ci-build-id#how-retrying-a-build-affects-ci-build-id

[TomaszG]

@bastienmoulia @agoldis for GitLab you can vote for https://gitlab.com/gitlab-org/gitlab/-/issues/195618

With such a variable, the director service could have a function to register what jobs are initial workers for the test run and differentiate them from the retried jobs where failed tests could be re-run.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.