Weird flicker in video for end to end test on updating to v10.10.0

[lidiagc]

Current behavior

We were previously using cypress version 9.7.0 and now want to update to version 10.10.0. However, we sometimes get an issue with the end-to-end tests when running on GitLab CI.

As you can see from the video, a weird flicker happens during the test and it eventually fails. Does it seem that the test is jumping up and down the commands as it was running two instances of the test? The video is only showing the start of the test and is pixelated since this is a private project.

weird-video-e2e.mp4

We haven't been able to reproduce this behavior locally, and it doesn't constantly happen on the CI. It is also not specific to this test, sometimes the other end-to-end tests fail occasionally and the videos have the same weird flicker. These tests didn't have this issue before upgrading cypress version.

Desired behavior

We would like to be able to update cypress to version 10, but this issue is delaying the update since we cannot rely on the end-to-end tests for our CI.

Test code to reproduce

We can't provide the full test code since this is a private project.

• cypress.config.ts

import { defineConfig } from "cypress";

export default defineConfig({
    chromeWebSecurity: false,
    defaultCommandTimeout: 15000,
    viewportWidth: 1920,
    viewportHeight: 1080,
    videoUploadOnPasses: false,

    projectId: "xxxxxx",

    e2e: {
        setupNodeEvents(on, _config) {},

        retries: {
            runMode: 0,
        },

        specPattern: "cypress/e2e/**/*.{ts,tsx}",
    },
});

• cypress-dockerfile

FROM cypress/included:10.10.0

RUN npm install -g pnpm@6.2.1

# other packages installed necessary to run our e2e tests

• gitlab-ci.yml

itest:end-to-end:
    image: ${ITEST_IMAGE}
    stage: itest
    script:
      # backend necessary scripts
  
      - cd frontend
      - echo -e "\e[0Ksection_start:`date +%s`:frontend-deps[collapsed=true]\r\e[0KInstalling frontend deps"
      - CYPRESS_INSTALL_BINARY=0 pnpm install --no-verify-store-integrity --frozen-lockfile -r
      - echo -e "\e[0Ksection_end:`date +%s`:frontend-deps\r\e[0K"
      - cypress run
        --config baseUrl="http://localhost:8001/",defaultCommandTimeout=60000,watchForFileChanges=false,pageLoadTimeout=100000,numTestsKeptInMemory=0
        --browser chrome
        --reporter mocha-junit-reporter
        --reporter-options 'mochaFile=cypress/reports/junit/junit-[hash].xml'
        --env KC_HOSTNAME="localhost:8080"
        --spec clusters-active.ts
    artifacts:
      when: on_failure
      paths:
        - frontend/cypress/screenshots
        - frontend/cypress/videos
        - logs
      expire_in: 15 days
      reports:
        junit: frontend/cypress/reports/junit/*.xml

• gitlab output

$ cypress run --config baseUrl="http://localhost:8001/",defaultCommandTimeout=60000,watchForFileChanges=false,pageLoadTimeout=100000,numTestsKeptInMemory=0 --browser chrome --reporter mocha-junit-reporter --reporter-options 'mochaFile=cypress/reports/junit/junit-[hash].xml' --env KC_HOSTNAME="localhost:8080" --spec clusters-active.ts
[10659:1017/234749.919499:ERROR:node_bindings.cc(279)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[10659:1017/234750.894445:ERROR:zygote_host_impl_linux.cc(263)] Failed to adjust OOM score of renderer with pid 10889: Permission denied (13)
libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[10889:1017/234750.909020:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.
====================================================================================================
  (Run Starting)
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        10.10.0                                                                        │
  │ Browser:        Chrome 100 (headless)                                                          │
  │ Node Version:   v16.14.2 (/usr/local/bin/node)                                                 │
  │ Specs:          1 found (clusters-active.ts)                                                   │
  │ Searched:       cypress/e2e/customer/clusters-active.ts                                        │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  clusters-active.ts                                                              (1 of 1)
Timed out waiting for the browser to connect. Retrying...
  (Results)
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        9                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      8                                                                                │
  │ Screenshots:  2                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 minute, 40 seconds                                                             │
  │ Spec Ran:     clusters-active.ts                                                               │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
  (Screenshots)
  -  /builds/.../frontend/cypress/screenshots/clusters-act     (1280x720)
     ive.ts/test -- before all hook (failed).png                        
  -  /builds/.../frontend/cypress/screenshots/clusters-act     (1280x720)
     ive.ts/test -- after all hook (failed).png                         
  (Video)
  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /builds/.../frontend/cypress/vid    (6 seconds)
                          eos/clusters-active.ts.mp4                                                
====================================================================================================
  (Run Finished)
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  clusters-active.ts                       01:40        9        -        1        -        8 │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 1 failed (100%)                     01:40        9        -        1        -        8  

Cypress Version

10.10.0

Node version

v16.14.2

Operating System

cypress/included:10.10.0

Debug Logs

No response

Other

Which debug logs would be more relevant for this? DEBUG=cypress:* has too many logs, so the GitLab job exceeds the maximum logs saved before the test finishes.

[mschile]

Hi @lidiagc 👋 , thanks for logging this issue.

Does it seem that the test is jumping up and down the commands as it was running two instances of the test?

Yes, it does look like that though I can't think of anything that would cause this type of behavior.

Does this happen on all browsers or just Chrome? If you run just a single spec does it happen? Does it also happen in Cypress 10.0.3?

[lidiagc]

Hi @mschile, thank you for replying!

We have in total 4 jobs that run our end-to-end tests:

• (1/4): runs only one spec file (the one I mentioned in the issue)
• (2/4): runs two spec files
• (3/4): runs two spec files
• (4/4): runs four spec files

First, I tested with version 10.10.0 and browser Electron:

• (1/4)
  The same issue happened at least once

• (2/4)
  The flicker happened on the first spec and the test failed
  The second spec passed

• (3/4)
  Similarly, the flicker happened on the first spec and the test failed
  The second spec passed

• (4/4)
  Again, the flicker happened on the first spec and the test failed
  The following three spec files passed

Our tests are not prepared to run with Firefox, so they all failed but not because of the flicker.

Then, I downgraded to version 10.0.3 and ran the tests a couple of times both with Chrome and Electron and the flicker issue never appeared. I'm guessing something after that version changed so the issue happens occasionally?

It's good to know that there is a version of Cypress 10 that doesn't have this issue; however, we wouldn't want to upgrade to this version because of the relative path issue that was fixed in version 10.9.0. Our component tests have a lot of similar names, and what differentiates them is their path. Upgrading to version 10.0.3 wouldn't be viable.

Let me know if there are other scenarios you want me to test!

[mschile]

Thanks for the update @lidiagc! I would love to narrow in on which Cypress version caused the regression. My initial guess is Cypress 10.8.0 may have broke it. Would you be able to try 10.7.0 and 10.8.0?

[lidiagc]

I tried versions 10.7.0 and 10.8.0 and the flicker never happened. Then, I upgraded to version 10.9.0 and it occurred. I suppose something in that version caused the regression?

[mschile]

@lidiagc, thanks for determining which version caused the regression. Unfortunately, I haven't been able to reproduce the flickering. I know you aren't able to provide a link to your private repository, but are you able to recreate the issue in a public one or possibly using the cypress-test-tiny project?

[ryanthemanuel]

Hi @lidiagc. We have a theory about things that might be causing video oddities. I created a 10.9 custom binary with some code tweaks to test out that theory. Can you try this binary to see if it works?

npm install https://cdn.cypress.io/beta/npm/10.9.1/linux-x64/10.9.0-minus-video-refactor-c2135e7e0e6b269a755e7f4309f90630af81d3b9/cypress.tgz

[lidiagc]

Hi @ryanthemanuel, thank you for looking into this. I installed the custom binary and ran the end-to-end tests on GitLab. The flicker still happens and it seems way more frequent than before.

We have recently upgraded to version 11.0.1 and we have encountered the flicker a couple of times, but it seemed way less frequent than in version 10, hence our decision to upgrade.

[mjhenkes]

@lidiagc, given that you're seeing this less frequently on 11.0.1 would you be ok with us closing this issue?

[chasemgray]

Hi @mjhenkes
I've been struggling to debug this issue for the last week or two. Based on the cypress debug logs we can see that cypress launches one version of chrome, but then fails to successfully connect to it. It then launches another version very quickly after that and then does successfully launch. Both instances of chrome are running in parallel and executing all the same tests. This causes duplicate logs, duplicate commands, and multiple dom snapshots which all get compiled together into a video which appears to be flickering but it's really just oscillating between the different browsers running the same tests. This still occurs on version of Cypress up to 12.2.0 and started around version 10.10.0. It doesn't occur for us on version 10.0.0.
We have been able to reduce the occurrences of this issue by increasing CYPRESS_INTERNAL_BROWSER_CONNECT_TIMEOUT to a very high value so that it eventually connects to chrome. Do we know why the newer version of Cypress fails to connect within the timeout period? And when it fails to connect, how come it isn't aware an instance of Chrome is running the tests the background while it launches another one?