Firefox 124 + Cypress 13.7.1 The browser never connected. Something is wrong. The tests cannot run. Aborting... after first test with multiple tests on CI

[akepley0280]

Current behavior

After upgrading to 13.7.1 to fix the Firefox 124 issue, we confirmed that local runs of open and run work to run tests in Firefox with multiple specs synchronously.
But, overnight on the Github CI workflow action, we are continuing to see failures on any jobs that have multiple specs for Firefox only. Our Github action uses @cypress/github-actionsv5 with the following

    runs-on: ubuntu-latest
    ...
      - name: Run firefox
        if: ${{ always() }}
        uses: cypress-io/github-action@v5
        with:
          working-directory: e2e-tests
          browser: firefox
          headed: true
          spec: 'cypress/specs/parallel/${{ matrix.spec-folder }}/*.js'

Output without debug showing the first test succeeding but the second test failing/aborting:

DevTools listening on ws://127.0.0.1:44811/devtools/browser/2334dc2a-af89-432c-88d2-b245882d6beb

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.7.1                                                                         │
  │ Browser:        Firefox 124                                                                    │
  │ Node Version:   v16.20.2 (/home/runner/runners/2.314.1/externals/node16/bin/node)              │
  │ Specs:          2 found (collections.js, knowledgebase-search.js)                              │
  │ Searched:       cypress/specs/parallel/knowledgebase/*.js                                      │
  │ Experiments:    experimentalMemoryManagement=true,experimentalStudio=true                      │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  collections.js                                                                  (1 of 2)

Your project has set the configuration option: `chromeWebSecurity` to `false`.

This option will not have an effect in Firefox. Tests that rely on web security being disabled will not run as expected.
Still waiting to connect to Firefox, retrying in 1 second (attempt 18/62)
Still waiting to connect to Firefox, retrying in 1 second (attempt 18/62)
Browserslist: caniuse-lite is outdated. Please run:
  npx update-browserslist-db@latest
  Why you should do it regularly: https://github.com/browserslist/update-db#readme


kbc-XOiAutomatedOrg-auto-new-collection
    ✓ Creates Collection (24342ms)
    ✓ Does Collection Stuff (18939ms)
    ✓ Deletes Collection ([85](https://github.com/xoeye/vision-web/actions/runs/8397901193/job/23001990647#step:9:86)63ms)


  3 passing (52s)

Warning: We failed capturing this video.

This error will not affect or change the exit code.

Error: Insufficient frames captured to create video.
    at ChildProcess.<anonymous> (<embedded>:2623:16306)
    at ChildProcess.emit (node:events:514:28)
    at Process.onexit (node:internal/child_process:291:12)

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        3                                                                                │
  │ Passing:      3                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     52 seconds                                                                       │
  │ Spec Ran:     collections.js                                                                   │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  knowledgebase-search.js                                                         (2 of 2)

Your project has set the configuration option: `chromeWebSecurity` to `false`.

This option will not have an effect in Firefox. Tests that rely on web security being disabled will not run as expected.
Warning: vkCreateInstance: Found no drivers!
Warning: vkCreateInstance failed with VK_ERROR_INCOMPATIBLE_DRIVER
    at CheckVkSuccessImpl (../../third_party/dawn/src/dawn/native/vulkan/VulkanError.cpp:[88](https://github.com/xoeye/vision-web/actions/runs/8397901193/job/23001990647#step:9:89))
    at CreateVkInstance (../../third_party/dawn/src/dawn/native/vulkan/BackendVk.cpp:458)
    at Initialize (../../third_party/dawn/src/dawn/native/vulkan/BackendVk.cpp:344)
    at Create (../../third_party/dawn/src/dawn/native/vulkan/BackendVk.cpp:266)
    at operator() (../../third_party/dawn/src/dawn/native/vulkan/BackendVk.cpp:521)


Timed out waiting for the browser to connect. Retrying...

Timed out waiting for the browser to connect. Retrying again...

The browser never connected. Something is wrong. The tests cannot run. Aborting...

The browser never connected. Something is wrong. The tests cannot run. Aborting...

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        0                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     0 seconds                                                                        │
  │ Spec Ran:     knowledgebase-search.js                                                          │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  collections.js                           00:52        3        3        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  knowledgebase-search.js                    0ms        -        -        1        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 2 failed (50%)                      00:52        3        3        1        -        -  

Error: Cypress tests: 1 failed

Chrome which runs on the same GH runner, same OS, using @cypress/github-actionsv5 has no issues

Desired behavior

Running Cypress in Github Workflow against Firefox 124 with @cypress/github-actionsv5 where multiple specs run synchronously works

Test code to reproduce

Should fail running against github action on FF124 using ubuntu image with multiple specs

Cypress Version

13.7.1

Node version

v16.20.2

Operating System

ubuntu-22.04

Debug Logs

cypress:server:socket-base socket-disconnecting transport close +99ms
  cypress:server:socket-base socket-disconnect transport close +0ms
  cypress:launcher:browsers firefox stdout: Stopped devtools server on 42925 +53ms
  cypress:launcher:browsers firefox stdout: 1711204983446	Marionette	INFO	Stopped listening on port 41353 +3ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35270 } +193ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35306 } +0ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35302 } +2ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35284 } +31ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35298 } +0ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35320 } +45ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51772 } +63ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51804 } +31ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35244 } +76ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51834 } +3ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35262 } +1ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51830 } +1ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35148 } +1ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 35134 } +1ms
  cypress:server:browsers:cri-client disconnected, starting retries to reconnect... { closed: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +1m
  cypress:server:browsers:cri-client disconnected, attempting to reconnect... { retryIndex: 1, closed: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +0ms
  cypress:server:browsers:cri-client connecting { connected: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +1ms
  cypress:server:browsers:cri-client could not reconnect, retrying... { closed: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede', err: Error: socket hang up     at connResetException (node:internal/errors:720:14)     at Socket.socketOnEnd (node:_http_client:525:23)     at Socket.emit (node:events:526:35)     at endReadableNT (node:internal/streams/readable:1359:12)     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { code: 'ECONNRESET' } } +16ms
  cypress:server:browsers:firefox-util firefox: reconnecting CDP +509ms
  cypress:server:browsers:cri-client closing cri client { closed: false, target: '62f586a6-876e-4bd5-b8dc-38bb9b96257a' } +5ms
  cypress:server:browsers:cri-client closed cri client { closed: true, target: '62f586a6-876e-4bd5-b8dc-38bb9b96257a' } +0ms
  cypress:server:browsers:browser-cri-client Attaching to target url about:blank +1m
  cypress:server:browsers:cri-client enqueueing command { command: 'Target.getTargets', params: undefined } +0ms
  cypress:server:browsers:cri-client disconnected, attempting to reconnect... { retryIndex: 2, closed: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +97ms
  cypress:server:browsers:cri-client connecting { connected: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +0ms
  cypress:server:browsers:cri-client could not reconnect, retrying... { closed: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede', err: Error: connect ECONNREFUSED 127.0.0.1:44279     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16) { errno: -111, code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 44279 } } +3ms
  cypress:server:browsers:cri-client disconnected, attempting to reconnect... { retryIndex: 3, closed: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +101ms
  cypress:server:browsers:cri-client connecting { connected: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +0ms
  cypress:server:browsers:cri-client could not reconnect, retrying... { closed: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede', err: Error: connect ECONNREFUSED 127.0.0.1:44279     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16) { errno: -111, code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 44279 } } +3ms
  cypress:launcher:browsers firefox exited: { code: 0, signal: null } +669ms
  cypress:server:browsers browser instance exit event received { code: 0, signal: null } +729ms
  cypress:server:preprocessor removeFile /home/runner/work/vision-web/vision-web/e2e-tests/cypress/specs/parallel/knowledgebase/collections.js +1m
  cypress:server:preprocessor base emitter plugin close event +0ms
  cypress:server:preprocessor base emitter native close event +1ms
  cypress:server:preprocessor base emitter native close event +1ms
  cypress:server:browsers:firefox closing remote interface client +15s
  cypress:server:browsers:cri-client not closing, cri client is already closed { closed: true, target: '62f586a6-876e-4bd5-b8dc-38bb9b96257a' } +17ms
  cypress:server:browsers:cri-client closing cri client { closed: false, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +0ms
  cypress:server:browsers:cri-client closed cri client { closed: true, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +0ms
  cypress:webpack close /home/runner/work/vision-web/vision-web/e2e-tests/cypress/specs/parallel/knowledgebase/collections.js +1m
  cypress:server:browsers:cri-client disconnected, not reconnecting because client is closed { closed: true, target: 'ws://127.0.0.1:44279/devtools/browser/00d49155-e24f-4458-80a1-76e2b78c3ede' } +85ms
  cypress:server:browsers browsers.kill called with no active instance +59s
Timed out waiting for the browser to connect. Retrying...
cypress:server:browsers browsers.kill called with no active instance +1m
The browser never connected. Something is wrong. The tests cannot run. Aborting...
  cypress:server:run onError +1m
The browser never connected. Something is wrong. The tests cannot run. Aborting...
  cypress:server:run received project end +1ms
  cypress:server:run received videoController { videoController: undefined } +0ms
  cypress:server:run spec results: { error: 'The browser never connected. Something is wrong. The tests cannot run. Aborting...', reporter: undefined, reporterStats: null, screenshots: [], spec: { absolute: '/home/runner/work/vision-web/vision-web/e2e-tests/cypress/specs/parallel/knowledgebase/knowledgebase-search.js', fileExtension: '.js', fileName: 'knowledgebase-search', name: 'knowledgebase-search.js', relative: 'cypress/specs/parallel/knowledgebase/knowledgebase-search.js' }, stats: { duration: 0, endedAt: '2024-03-23T14:46:03.416Z', failures: 1, passes: 0, pending: 0, skipped: 0, startedAt: '2024-03-23T14:46:03.416Z', suites: 0, tests: 0 }, tests: [], video: null } +1ms
  cypress:server:run execute after:spec +0ms

Other

No response

[MikeMcC399]

@akepley0280

I tried to reproduce your issue using the repo https://github.com/cypress-io/cypress-example-kitchensink. The workflow I set up to use Firefox failed with Firefox 124.0 until I updated to Cypress 13.7.1 when it then succeeded, and so unfortunately I was not successful in recreating your problem.

I have a couple of comments which may or may not help:

• It looks like you have video explicitly enabled, and that does not work for current versions of Firefox (see Video functionality is not working on Firefox 93 and above #18415). I would suggest disabling it, which is the default in Cypress 13.x anyway (see https://docs.cypress.io/guides/references/migration-guide#Video-updates).

• cypress-io/github-action@v5 is no longer supported, as it relies on Node.js 16 which is in end-of-life status. You might like to try updating to cypress-io/github-action@v6 although I doubt that will make any difference.

• You also have the error from Warning: vkCreateInstance: Found no drivers! warning shown when running Cypress #29085 in your logs. This is probably irrelevant though.

[gsouf]

#29172 was supposed to fix this issue on cypress 13.7.1. After upgrading to firefox 124 and cypress 13.7.1 I am still experiencing this issue in my local setup and in CI pipelines:

  (Run Starting)

  ┌────────────────────────────────────────────────────────
  │ Cypress:        13.7.1                                                                         │
  │ Browser:        Firefox 124 (headless)                                                         │
  │ Node Version:   v20.10.0 (/usr/local/bin/node)                                                 │

First test passes then second tests fails to connect to the browser:

Timed out waiting for the browser to connect. Retrying...

Timed out waiting for the browser to connect. Retrying again...

The browser never connected. Something is wrong. The tests cannot run. Aborting...

The browser never connected. Something is wrong. The tests cannot run. Aborting...

I'll try to provide a minimal reproducible sample

[gsouf]

@MikeMcC399 May I request you to try again with firefox 124 and cypress 13.7.1 and try to specify a custom user agent in the cypress config file, ie:

module.exports = {

  // ... some config

  e2e: {
        // ... some config
  
        // specify user agent:
          userAgent: 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/114.0'
  }
}

For me it fails with the user agent string, and works when I remove it

[MikeMcC399]

@gsouf

Your Firefox user agent string is not consistent with Firefox 124.0. It should end with rv:124.0) Gecko/20100101 Firefox/124.0 if you have userAgent specified and you are testing with Firefox 124.0.

[MikeMcC399]

• The fix in PR fix: patch new tab creation for firefox 124 and up to fix issue where… #29179 relies on the user agent string in order to decide whether to do things differently for Firefox 124 and above.

[MikeMcC399]

@akepley0280

Are you also specifying userAgent in your tests?

Edit:
Define environment variable DEBUG=cypress:config:browser before running tests to find out if the log shows:

userAgent: null,
...
userAgent: { value: null, from: 'default' },

or something non-null.

[gsouf]

@MikeMcC399 from my expectation the user agent string itself shouldn't change how cypress behaves. We use it to make sure our app will detect that it's an unsupported browser during E2E, and I'm sure that other users will do the same. It's a lot faster than having to manage multiple firefox binaries when all we need is to change the UA string.

I'm not even sure why cypress would allow us to change the UA string if it breaks cypress itself.

Any chance we can expect a more solid support for firefox >124?

[MikeMcC399]

@gsouf

Thanks for your explanation!

• Your question would need to go back to the Cypress.io team who provided PR fix: patch new tab creation for firefox 124 and up to fix issue where… #29179. The author was @AtofStryker, so perhaps he will be able to comment why the version from the userAgent is being used instead of the version extracted from the Firefox binary, and whether this could be changed to suit also your use-case?

[akepley0280]

@MikeMcC399 I can definitely say we use a specific userAgent in our FF tests as well, making it break for us as well, only when running against CI (only because we only set this in the GH action workflow for the FF run)

'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:107.0) Gecko/20100101 Firefox/107.0'

@gsouf thanks for bringing this up! This was driving me insane

[akepley0280]

looking at our history, we can probably just update the userAgent used to 124.0 in the custom config. Our issue appeared to be something around the (X11; Ubuntu; Linux x86_64 that the CI runner uses Ubuntu. I'm going to verify if that works.

[AtofStryker]

@gsouf

Thanks for your explanation!

• Your question would need to go back to the Cypress.io team who provided PR fix: patch new tab creation for firefox 124 and up to fix issue where… #29179. The author was @AtofStryker, so perhaps he will be able to comment why the version from the userAgent is being used instead of the version extracted from the Firefox binary, and whether this could be changed to suit also your use-case?

I opted to use the userAgent in the fix because the fix is within Cypress' Firefox web extension. The extension is responsible for creating/deleting tabs in the Firefox browser, and the most available property to determine the version is the userAgent from that context. AFAIK, we don't have another way to immediately determining the version from within the browser context.

We could likely pass in the binary version detected in the app at time of launch and forward that to the extension request, but that seemed somewhat involved and unnecessary. It can be done, but it would need to be baked into Cypress itself and it likely a lot easier to adapt your UA to include the version in which is being run.