Handle test retries more intelligently #686
Replies: 2 comments 1 reply
-
Good point - it seems like a common problem with the existing orchestration implementation. So far I haven't been able to implement a generic solution that would support all CI providers, and groups, and still have a valid parallelization. One idea I had was to introduce a "--rerunId" flag - a run created for an existing rerunId would be considered a rerun and would run only the failed tests. That solution requires a more intrusive integration with cypress runner itself, which I was trying to avoid. I am hoping to get to it within the next 1-2 months. |
Beta Was this translation helpful? Give feedback.
-
I don't know the internal workings well enough, but I am assuming the director passes spec files to agents with the same ci-build-id until all matching specs have been distributed to agents. Would it make sense that if an agent subsequently requests specs with the same ci-build-id the director just passes it a previously failed spec file? I assume at the moment it just says there are no spec files left to test. |
Beta Was this translation helpful? Give feedback.
-
What is this feature about (1 sentence)
If an individual test fails in a parallel run the only option is to increment the identifier and run all tests again. Would be better to be able to run just failed tests again.
Why is it needed? What is the value? For whom do we build it?
Similar in requirement to #197 but if I could lay out my case in more detail... we run 8 agents in parallel and occasionally a test will fail. We already have retries within the cypress test run but the issue will occasionally be environmental and can therefore be fixed outside of the test run. At the moment we increment the $ATTEMPT number as well as $BUILD number. That means if I retry the stage the entire suite of tests runs again. If I rerun just the failed stage essentially the same thing happens but it happens on a single agent (the one that failed - but that agent gets given all the specs.
Ideally (and I don't know if cypress handles this so it might deviate from the baseline) we would be able to pass the original ci-build-id back to the director (along with a retry indicator?) - the director would decide that that run id has completed and pass out the failed specs again.
Beta Was this translation helpful? Give feedback.
All reactions