Configurable behavior for workers with a completed ciBuildId

[boxofcrates]

• I have searched issues for similar feature requests

What is this feature about (1 sentence)

Configurable behavior when a worker passes the director a ciBuildId corresponding to a run that has already completed. This will close a 'loophole' allowing CI pipelines to pass even when Cypress tests fail.

Why is it needed? What is the value? For whom do we build it?

I'll explain the issue with the current behavior. I'm using Gitlab as an example, but I think this applies to other scm systems:

Setup: CI pipeline running 3 parallel Cypress workers talking to sorry-cypress. One of the specs failed, so Gitlab shows cypress-2 job as failed, but cypress-1 and cypress-3 as succeeded.
Scenario: developer sees cypress-2 job failed, clicks retry on that job

cypress-2 job will start with the same ciBuildId. It connects to director, is told that all specs have already been run, and successfully quits.

However... the CI pipeline now shows as succeeding, with all jobs passing. Even though the one spec failed and was never re-run. There's no way to change this behavior from the Gitlab side, as there is no env var that a) is the same across all jobs in an initial pipeline run and b) is different when a job is re-tried.

High-level feature overview

I'm not sure what the best solution here is, but my thinking is:

• Configurable behavior in director when a worker comes in with a ciBuildId corresponding to an already-finished job
• Depending on configuration, either: PASS pass with no specs (current behavior), FAIL fail, or RE-RUN re-start the whole run and give all specs to the new worker

RE-RUN is IMO the ideal, but even just FAIL would allow closing this loophole.

@agoldis thoughts? If we can come up with an agree-able solution I'm happy to work on this, it'll save me from tearing my hair out in my professional life 😆

Other

[agoldis]

@boxofcrates thanks for opening that issue!

I tend to try using native cypress "retries" instead of complex rerun logic, see #197

So basically it's all about re-running the failed specs. Specifically, re-running only the failed specs. Let's try to break it down:

Rerunning all the specs

If you manage to generate a new ciBuildId when rerunning, it will just run all the specs. cypress agent generates ciBuildId automatically for retries. See cypress-io/cypress#8451. If you generate ciBuildId manually, you'd follow the the same path to make sure rerunning creates a new ciBuildId.

Rerunning specific failed specs

That should be possible to implement in theory, but would be quite cumbersome. Basically we need to be able to recognize if it's a rerun and fetch specific failed specs... plus support grouping etc. I'd try avoiding this complexity.

In general, if you use "retries" and tests still fail, and then you want another "retry" via CI, something is wrong with the tests :)

[carlos0202]

Hi guys,

Sorry for jumping into this discussion with my own request or help. In my case, I would like to be able to rerun all the specs from circleci no matter if they all passed or failed. I was trying to create a dynamic environment variable on circle-ci (holding current date-time) to add that variable to ci-build-id to produce different build ids per run with no success :'(.

Can anybody help me on this? Thanks in advance!!!

[agoldis]

@carlos0202 try using --ci-build-id $CIRCLE_WORKFLOW_ID-$CIRCLE_BUILD_NUM, that worked for me

[carlos0202]

@carlos0202 try using --ci-build-id $CIRCLE_WORKFLOW_ID-$CIRCLE_BUILD_NUM, that worked for me

Well, this could help me but I'll be facing another issue because I have multiple cypress/run steps in my circleci config. Thanks for the help, I need to figure out how can I solve this issue.

[akociemba]

@agoldis about this

In general, if you use "retries" and tests still fail, and then you want another "retry" via CI, something is wrong with the tests :)

It might be an issue with infrastructure, resolved outside of the pipeline, and the user might not want to create a new pipeline, just to retry few tests (especially in the case of CI/CD deployment pipelines). In such a case, it's really useful to be able to retry only failed tests (or even the whole test suite) instead of creating a new pipeline.

A special case of the above is running tests in parallel, for example, 4 threads, and one of the threads fail. Currently retrying failed job (with default ciBuildId in case of gitlab = pipelineId) is successful without actually retrying the tests, making the pipeline false-positive. Using combined ciBuildId like pipelineId_jobId, messes with smart spec division, as in our example we have now 4 runners running all the same, undivided specs, wasting our time.

[agoldis]

@akociemba thanks, those are useful use-cases indeed.

It's going to be quite a challenge in terms of implementation, also, it is different from the original cypress dashboard behaviour, so we'd thunk carefully about the implementation and implications.

[rszalski]

@agoldis another (hopefully legit) use-case for re-running the entire set of tests on failure, directly on CI.

We use Travis CI which runs 2 steps: first one (Build) builds the app and shares the artefacts through cache with the second step (e2e), which runs e2e tests as 4 parallel jobs:

This way the e2e jobs don't have to waste time building the app 4x which saves time on retries.

Now, it sometimes happens that test(s) fail (due to network/machine issue, backend problems, or simply due to flake). What we've been doing so far in this case (using the original Cypress Dashboard) was re-running those 4 parallel jobs to re-run the entire test suite fairly quickly. However, using sorry-cypress this is not working as described in OP's comment.

I'm trying to workaround this by combining some Travis CI ENV vars under --ci-build-id so that we get a) parallelism across 4 runs AND b) re-run of all the specs when one or more of those 4 jobs is re-run. So far it's tricky because the ENV vars are either stable for the entire build or change with every job, which prevents parallelism/grouping.

[agoldis]

@rszalski Curious how that worked on the original cypress dashboard - afaik the have the same behaviour and do not allow to "reuse" the same cibuildId. Would you share your setup / documentation / screenshot from the original Cypress dashboard to give me a hint?

Also, I figure that the build step could probably be cached to avoid rebuilding for the same commit....

You can achieve a similar behavior using recent @Aeolun work that allows explicitly "removing" a spec from a run: #399

You run the tests, remove the spec files that have failed, then rerun the tests again with same ciBuildId. In theory, this can be automated - i.e. rerunning from CI can trigger the flow mentioned.

I am hesitant to embed this behavior by default because it's quite hard to implement and maintain and also would create confusion among existing users.

[rszalski]

Curious how that worked on the original cypress dashboard - afaik the have the same behaviour and do not allow to "reuse" the same cibuildId.

@agoldis this is a very good question and one I wanted to answer myself. Here are my findings.

I switched back to using the original cypress dashboard (if the alternative is sorry-cypress, is this one unapologetic-cypress? :D ). I tested 2 scenarios: first without passing any ciBuildId and second with passing ciBuildId set explicitly to TRAVIS_BUILD_ID , which is what Cypress Dashboard seems to do anyway (!).

For both of these scenarios I pushed a new commit (triggering a completely new build) and then re-tried all the 4 jobs twice (retaining identical TRAVIS_BUILD_ID across those 2 re-runs).

Example of a first scenario (3 full e2e test runs):

Here are the details of these 3 runs. Thankfully cypress Dashboard surfaces the ciBuildId.

First run (new build, triggered with commit push). Notice the ciBuildId contains a correct TRAVIS_BUILD_ID with some Cypress-internal prefix (travis-), which is stable and we can disregard it.

Second run (first retry through Travis UI). Notice that we now have what looks like a UUID added as a suffix to the ciBuildId (highlighted on the screenshot). I believe this is the mechanism through which Cypress Dashboard ensures that retried test runs for the same build ID get a different ciBuildID hash/key.

Third run (second retry through Travis UI). Again, UUID appended, different each time, for "cache" busting.

Identical thing happened in the second scenario, when I explicitly provided the ciBuildID=$TRAVIS_BUILD_ID. First run had travis-${TRAVIS_BUILD_ID} ciBuildId, while subsequent reruns generated travis-${TRAVIS_BUILD_ID}-${UUID}.

I also went through the client code of Cypress but did not find (or missed?) logic that is responsible for the generation of the ciBuildId. The only thing the client (runner) library is concerned with is whether --ci-build-id has been passed or not. If it hasn't, it is still ok if the CI is one of those when that ID can be inferred from. Otherwise it errors out. This seems to be confirmed by watching requests made to sorry-cypress' director service:

(note that this is a random build #)

...
ci: {
  params: {
    travisBuildId: "240490265"
    travisBuildNumber: "17297"
    travisBuildWebUrl: "<repo url>"
    travisCommitRange: "a829b364be4b...babb58d8fa691a9798"
    travisEventType: "pull_request"
    travisJobId: "544911148"
    travisJobNumber: "17297.3"
    travisPullRequest: "<pr num>"
    travisPullRequestBranch: "<branch>"
    travisPullRequestSha: "<sha>"
    travisRepoSlug: "<repo>"
  }
  provider: "travis"
}
ciBuildId: null
...

I think that ultimately it is Cypress Dashboard that infers the correct ciBuildId when it's missing but also when it is provided. It handles the "UUID" appending so that re-runs are correctly recognised (and get a fresh set of specs to re-run). It might be simply waiting until the timeout for registering machines for a parallel run, and using a different UUID next time anyone requests tests within that timeout.

@agoldis do you think something similar would be possible to do in sorry-cypress? Or maybe there is a way to hack around this for now? I think it has to be implemented at a director-level, because if CI jobs are truly separate and run in parallel, I don't think they can come up with a reliable method of grouping themselves if CI doesn't provide that explicitly as an ENV var.

The only horrible hacks that come to my mind is for those parallel CI jobs to come up with a date-based hash. They could round current date time to let's say 10 minutes and assuming all those machines would register with director within that 10 minutes they would be considered part of the same group. If they miss the window (due to queue congestion on CI) they run the entire thing from scratch (unideal but maybe unavoidable?). It feels hacky and undeterministic but still better than false positives on re-runs (not re-running tests).

Also, I figure that the build step could probably be cached to avoid rebuilding for the same commit....

That would be a nice to have, but not a priority for now, at least for our team. Currently we are only concerned with getting test re-runs to work with parallelism enabled (it is easy to get re-runs that run the entire test suit by providing an ever-changing ciBuildId but this disables parallelism).

[agoldis]

Thanks for the detailed write up @rszalski! Now I see that I misinterpreted your original question.

You are right:

• re-running a CI pipeline should create a new run
• the dashboard is responsible for generating ciBuildId unless is not provided explicitly via a CLI flag

Here's the current implementation of ciBuildId generator - it's quite naive as you can see :)

So, you're right again, the challenge is just to find the right combination of ci.params, or as you've described:

...combining some Travis CI ENV vars under --ci-build-id so that we get a) parallelism across 4 runs AND b) re-run of all the specs when one or more of those 4 jobs is re-run.

Looking at the original dashboard ciBuildId structure travis-${TRAVIS_BUILD_ID}-${UUID} it seems like they were able to capture "reruns" via UUID suffix.

Let's try to capture the requests for:

• original run
• rerun

and compare the differences.

There must be some parameter that's different between those - we should be able to use it for creating a different ciBuildId.

[rszalski]

There must be some parameter that's different between those - we should be able to use it for creating a different ciBuildId.

Hopefully that is the case and it should be easy to verify. It's easy for me to compare request for sorry-cypress director, because I have access to GCP logs where I deployed the director. However, I'm unsure how to best capture the traffic between the CI job and the "original" Cypress Dashboard. @agoldis what would be the best method? Running tcpdump to get the .pcap? What did you use for sniffing network? I wan't to pick the best approach before I spend 5h re-running CI jobs while testing 😭

[agoldis]

oh, you can use sorry-cypress for that, the requests are exactly the same

[rszalski]

oh, you can use sorry-cypress for that, the requests are exactly the same

Right! I somehow imagined those would differ, but cy2 is a thin wrapper and doesn't alter the network protocol. Thank you for the hint @agoldis

I ran another experiment, same setup but only 2 builds (first one is from a commit push, second is from me retriggering the jobs in Travis UI). Each build had 4 workers executing e2e specs (8 logs in general).

I've exported the GCP Cloud Run logs, sorted the JSON keys and removed some sensitive info (like app keys), but ensuring first that this info was identical across all 8 logs.

I'm attaching 2 files for comparison. They correspond to the first and second run of Machine # 2. Second run is a re-run triggered through UI.

sorry_cypress_2_logs.zip

Here is the diff between those files:

$ diff sorry_cypress_anon_machine2_build1_rerun*
2c2
< 	"insertId": "617a911000087255462ab90e",
---
> 	"insertId": "617a9c2a000052acf3800d42",
32c32
< 		"content-length": "5650",
---
> 		"content-length": "5648",
34c34
< 		"forwarded": "for=\"35.237.212.185\";proto=https",
---
> 		"forwarded": "for=\"35.196.82.30\";proto=https",
48c48
< 						"idle": 51370,
---
> 						"idle": 56620,
50,52c50,52
< 						"nice": 300,
< 						"sys": 18080,
< 						"user": 48760
---
> 						"nice": 0,
> 						"sys": 17890,
> 						"user": 47780
59c59
< 						"idle": 52260,
---
> 						"idle": 51740,
61,63c61,63
< 						"nice": 100,
< 						"sys": 18080,
< 						"user": 46600
---
> 						"nice": 240,
> 						"sys": 17530,
> 						"user": 46990
68c68
< 				"free": 400437248,
---
> 				"free": 511594496,
76c76
< 		"requestId": "30b62d0c-251f-4f7e-b2cd-a78d293162f6",
---
> 		"requestId": "ad746f3b-4747-46cd-93e4-a19657e4646b",
89,91c89,91
< 		"time": 1635422480552,
< 		"traceparent": "00-25e3e3bdb885505d25c127f5eedbf756-0b9eb8f99b2c2d3d-01",
< 		"x-cloud-trace-context": "25e3e3bdb885505d2c127f5eedbf756/837309962927222077;o=1",
---
> 		"time": 1635425322019,
> 		"traceparent": "00-17553e7746909d0a0087a21af51bc822-3b84180c1e1906b9-00",
> 		"x-cloud-trace-context": "17553e7746909d0a0087a21af51bc822/4288579185487185593",
94c94
< 		"x-forwarded-for": "35.237.212.185",
---
> 		"x-forwarded-for": "35.196.82.30",
103c103
< 	"receiveTimestamp": "2021-10-28T12:01:20.555052541Z",
---
> 	"receiveTimestamp": "2021-10-28T12:48:42.281640490Z",
114c114
< 	"timestamp": "2021-10-28T12:01:20.553557Z"
---
> 	"timestamp": "2021-10-28T12:48:42.021164Z"

Most of the differing values are from GCP tracing. The entire commit/CI portion of the payload is identical. The only reasonably-formatted value that could be used to differentiate the runs is the requestId, but it is always different, so it would nullify parallelism.

So overall, I still think that it is actually Cypress Dashboard that has some additional logic to come up with the UUID portion of the key to allow for parallel-ized re-runs.

Right now I cam up with a workaround (not as elegant as what Cypress Dashboard probably implements, but gets the job done). I generate a timestamp that is basically rounded to 10 minutes allowing me to have both parallelism and correct re-runs (assuming the suite runs longer than 10 minutes!).

a=$(date +%s); export ISO_DATE_MODULO=$(date -ud "@$((a-a%(10*60)))" --iso-8601=seconds)
SORRY_CYPRESS_BUILD_ID="$TRAVIS_BUILD_ID-$ISO_DATE_MODULO"