Percy adds 5+ mins to CI run time

[mncharlton]

Hi,

When running our Cypress tests in CI (Jenkins), they take around 12-14mins to run, if I also run Percy as part of these tests then that time consistently goes up to 19-21mins.
It isn't the taking of individual snapshots that makes the difference (I've tried with 30, 10 and 1 snapshots and they all add the same time to the test run), suggesting it's something else Percy is doing - is there anything we can do to make this faster, we really don't want to increase our run time by almost 50%...

I emailed support the question above who asked me to post here with the following information:
- Do you have a lot of allowed hostnames added? No, we don't have any added.
- What's the network idle timeout set to? We haven't specified it, so it's whatever the default it.
- Do you have a lot of widths passed? No, we don't pass any specific widths.

As you can probably guess from those answers 😄 Our configuration is extremely simple, the percy.yml is simply:

version: 2
snapshot:
  percyCSS: |
    [data-test="user.avatar"] {
      visibility: hidden !important;
    }

Thanks!

[Robdel12]

Hey @mncharlton! This sounds interesting. Can you provide full debug or verbose logs (unedited)? The logs provide timings between each action so we'll be able to see what exactly is slow: https://docs.percy.io/docs/debugging-sdks#debug-vs-verbose-logging

[mncharlton]

Hi @Robdel12, thanks for getting back to me. For context, our Cypress CI runs on two machines, with each machine running two instances of Cypress so we end up with 4 sets of logs that I've attached below.

Nothing jumps out to me as taking a huge amount of time, but I might be missing something, one thing I've noticed, that I don't know if they're relevant or not is that as we have two instances running on each machine when Percy tries to start on the second instance it fails (see verbose0.2 for an example), this may be a separate problem, but is it possible to run Percy twice on one machine like this? (or have two instances of Cypress share one instance of Percy).

For an idea of the difference in time, this is how long this run took:

And this is a "normal" run (where the only difference is that we don't use Percy):

Verbose Logs (The only snapshot in this run is in verbose0.1, corresponding Percy build: https://percy.io/37d22c48/percy-pr/builds/16231702/unchanged/914136074?browser=safari&viewLayout=overlay&viewMode=new&width=1526):

Verbose0.1.txt
Verbose0.2.txt
Verbose1.1.txt
Verbose1.2.txt

Hopefully you can shed some light on this! 😄

[Robdel12]

Looking at the logs it doesn't look like Percy is actually running (port conflict errors). But I do see there's a lot of time spent downloading the browser across each machine:

00:07:22.726  [percy] Downloading Chromium 885264...
00:07:35.486  [percy] Successfully downloaded Chromium 885264

I would setup the browser to download with your dependencies and then persist that across the workflows: https://docs.percy.io/docs/caching-asset-discovery-browser-in-ci That way the browser is downloaded once and that should cut the CI time you're seeing down to what it used to be.

As for percy exec:start and --parallel, you totally can do that. All the -parallel flag is doing under the hood is setting PERCY_PARALLEL_TOTAL to -1. So you can set that env var and then call your finalize at the end of the test run. https://docs.percy.io/docs/parallel-test-suites#how-it-works

[mncharlton]

Thanks @Robdel12! Downloading with the dependencies looks like it's made a huge difference, and using the exec:start to start a local percy server before running the two instances of Cypress has sorted those conflict errors out.
The time it appears to be adding to the runs is down to something much more acceptable now (<1min) with the setup and taking the snapshots.
I think we're looking pretty good - thanks for your help!

[mncharlton]

For the one instance failing to start, I had a re-read of https://docs.percy.io/docs/parallel-test-suites#introduction and I see there is a way to just start one Percy server on a machine and then share that amongst test runs. Is there a way to combine that with the --parallel flag so that the runs on the different machines can then be combined? It doesn't look like --parallel is available on percy exec:start