Cypress 13 - GLIBC_2.29 not found (addon-native) - preventing opening cypress

[josephearl]

Current behavior

Our Jenkins pipelines using Cypress 13.2.0 were working fine this morning.

Today a Cypress update 13.3.0 was released. With 13.3.0 Cypress pipelines fail with an error:

Opening Cypress...
DevTools listening on ws://127.0.0.1:42471/devtools/browser/31a66146-18bd-478a-b7cc-acbae92826e7
/lib64/libm.so.6: version `GLIBC_2.29' not found (required by /home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node)
Error: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node)
    at process.func [as dlopen] (node:electron/js2c/asar_bundle:2:1869)
    at Module._extensions..node (node:internal/modules/cjs/loader:1354:18)
    at Object.func [as .node] (node:electron/js2c/asar_bundle:2:1869)
    at Module.load (node:internal/modules/cjs/loader:1124:32)
    at Module._load (node:internal/modules/cjs/loader:965:12)
    at f._load [as origLoad] (node:electron/js2c/asar_bundle:2:13377)
    at PackherdModuleLoader.tryLoad (evalmachine.<anonymous>:1:741495)
    at Function.<anonymous> (evalmachine.<anonymous>:1:751709)
    at d._load (<embedded>:4510:29351)
    at Module.require (node:internal/modules/cjs/loader:1148:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.H (<embedded>:4562:137346)
    at s.exports (<embedded>:4562:143111)
    at <embedded>:4569:3280
    at tryCatcher (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/util.js:16:23)
    at Function.<anonymous> (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/method.js:39:29)
    at Object.runElectron (<embedded>:4569:3050)
    at Object.startInMode (<embedded>:4569:5738)
    at <embedded>:4569:4826
    at tryCatcher (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at PromiseArray._resolve (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise_array.js:126:19)
    at PromiseArray._promiseFulfilled (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise_array.js:144:14)
    at Promise._settlePromise (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at Promise._settlePromise (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:582:21)
    at Promise._settlePromise0 (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at Promise._settlePromise (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:582:21)
    at Promise._settlePromise0 (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at PromiseArray._resolve (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise_array.js:126:19)
    at PromiseArray._promiseFulfilled (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise_array.js:144:14)
    at Promise._settlePromise (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at /home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/nodeback.js:42:21
    at <embedded>:2086:115512

I reverted to 13.2.0 and still get the same error.

I had to revert back to 12.x to get things working

Desired behavior

It should work with RHEL 8

Test code to reproduce

Fails before running any tests

Cypress Version

13.3.0

Node version

v18.15.0

Operating System

RHEL 8
Linux x86_64

Debug Logs

No response

Other

No response

[ahayes91]

What is the resolution for this issue, is #27902 the best reference? Thank you!

[josephearl]

I closed because reverting to 13.2.0 was still failing and I thought it was a problem/change on our CI, but reopening because it does seems like a Cypress change/issue

[jason-stoudt]

Another data point for this problem:

Application Data: /home/githubrunner/.config/cypress/cy/development
Browser Profiles: /home/githubrunner/.config/cypress/cy/development/browsers
Binary Caches: /home/githubrunner/.cache/Cypress

Cypress Version: 13.1.0 (stable)
System Platform: linux (Amazon Linux - 2 (Karoo))
System Memory: 8.16 GB free 6.77 GB

cypress-io/github-action@v5

This was working prior to yesterday.

[floifymatias]

More info

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"

cypress-io/github-action@v6

This too was working yesterday.

[jennifer-shehane]

We're actively investigating this issue. Thank you for your patience.

[MikeMcC399]

I can reproduce this issue on Ubuntu 18.04.6 LTS with Cypress 13.0.0 and later, however I needed to use the now unsupported Node.js v16.20.2. npx cypress run immediately fails. It is not related to recording in Cypress Cloud.

Reverting to Cypress 12.17.4 works around the issue. For Ubuntu this is however not a good long-term resolution since it relies on the unsupported Node.js v16.20.2. For Ubuntu 18, the better strategy would be to update to Ubuntu 20 or 22 and use either Node.js 18 or 20.

[jennifer-shehane]

The Cypress provided docker-images using Node .js 18 and later are on GLIBC 2.31. You should be able to use our docker-images and not encounter this error. https://github.com/cypress-io/cypress-docker-images

We're investigating possible fixes for users encountering this that are not using our docker-images

[MikeMcC399]

The Cypress provided docker-images are on GLIBC 2.31.

That would apply to Cypress Docker images using Node.js 18 and later.

Some earlier Cypress Docker images based on Node.js 16 (which is end-of-life and no longer supported by Cypress) such as

• cypress/base:16.0.0
• cypress/browsers:node16.5.0-chrome94-ff93

are showing GLIC 2.28 (ldd --version) and are based on Linux 10 (buster) (cat /etc/os-release)

So this would also be something to check for Cypress Docker users wanting to run Cypress 13.x that they should use a Cypress Docker image with a minimum Node.js 18 version. This is the earliest version of Node.js which is currently supported.

[tkharuk]

@jennifer-shehane we use cypress/base:16 and it worked well with Cypress v13 for a couple of days. Upgrading to cypress/base:18 is not straightforward and would require some changes in our infrastructure.

So we will have to either wait for your rollback of whatever happened yesterday or downgrade to Cypress v12 for a time being :(

[jennifer-shehane]

We're still actively working through solutions. Downgrading Cypress to 12.17.4 is the quickest fix in the meantime of us fixing the issue. Using the Cypress docker-image that has Node.js v18 is another solution.

We'll update this thread when we have more updates.

[jennifer-shehane]

We’re hoping to have a fix out by tomorrow. 🤞

[baus]

Just to reiterate Jennifer's statement above, we have determined the source of the problem and are working to prepare an updated release tomorrow, Sept 29.

[MikeMcC399]

@tkharuk

we use cypress/base:16 and it worked well with Cypress v13 for a couple of days. Upgrading to cypress/base:18 is not straightforward and would require some changes in our infrastructure.

You mentioned using cypress/base:16 whereas in the other issue (#27902 (comment)) you mentioned using cypress/base:16.13.2. Perhaps you didn't mean the specific Docker tag cypress/base:16, because this is in fact equivalent to cypress/base:16.14.2-slim, which uses the later GLIB 2.31.

Although Node.js 16 is no longer supported, you may be able to just update to a later cypress/base:16.* Cypress Docker image version which is based on Linux 11 (bullseye) with GLIB 2.31 installed instead of the earlier cypress/base:16.* versions which used Linux 10 (buster) with GLIC 2.28. The cypress/base CHANGELOG seems to be no longer maintained and it's not quite accurate anyway, however it looks like the version cypress/base:16.13.2 was the last one to use the problematic Linux 10 (buster) version.

The latest cypress/base version based on Node.js 16 is cypress/base:16.18.1-with-git.

You could either wait for the promised fix in Cypress 13.x or you might like to try out using the Cypress Docker image cypress/base:16.18.1-with-git. In the long term you should be aiming to use Node.js 18 or 20 as I think you already understand.

[baus]

We have updated the release to address this issue. We are considering this issue to be resolved. If you continue to see errors related to glibc, please report here and reopen the issue.

[MikeMcC399]

@baus

We have updated the release to address this issue. We are considering this issue to be resolved. If you continue to see errors related to glibc, please report here and reopen the issue.

Can you explain how you have updated the release? Which version(s) have been updated?

Do you have any instructions about how to update an existing installation? For instance would it be necessary to run

npx cypress install --force

or similar?

[jennifer-shehane]

@MikeMcC399 13.0+ versions should be fixed. If you're not immediately seeing the fix, you could run cypress cache clear then reinstall.

[tkharuk]

can confirm that our "nightly" workflows that been failing, are green now.
thanks everyone for a quick fix 👍🏼

[MikeMcC399]

So I guess the fix was to replace the cypress.zip binaries on https://download.cypress.io?

That will be a problem for CI workflows like cypress-io/github-action which do caching. Admins will need to manually delete GitHub Actions cache files on GitHub. On the monorepo https://github.com/cypress-io/github-action there are 22 cache files (one for every example folder) which cache the Cypress binary, that would have to be deleted by hand.

Please consider building a new release such as 13.3.1 for such situations. Then the process is the standard one for updating to a new Cypress version. That would allow a new Cypress version to be written to a lockfile (npm / Yarn) which would then have a new hash and that would cause the cache to be invalidated and rewritten.

[jennifer-shehane]

@MikeMcC399 Due to the nature of the issue, this fix is outside of the standard process for updating. We don't currently have plans to do a patch release at this time outside of our regular release cadence.

Thanks for pointing out the unique caching for github-actions. Many CI workflows do have unique caching strategies. We'll keep following up on this issue, if there are issues with clearing the caching we'll try to assist. We don't want this to be a blocker for people.

[JessefSpecialisterren]

I'd like to second @MikeMcC399 's plea for doing a new release for updates like this. We're having to jump through several hoops here to avoid failed runs due to cached Cypress binaries:

• Our CI environment does not have direct internet access, meaning the Cypress binary is obtained through a URL-specific proxy outside of our control with an unknown caching policy
• Additionally, our CI runs happen on a randomly selected machine from a pool, each of which has their own cache. This means we effectively have to clear the cache and redownload the binary before every CI run until a new Cypress release comes out, because we cannot guarantee that there are no outdated binaries still cached somewhere in that machine pool

Doing a new release for this change would have saved us a lot of hassle

[MikeMcC399]

@JessefSpecialisterren

I'd like to second @MikeMcC399 's plea for doing a new release for updates like this.

Hopefully this will be a one-off situation. Nevertheless a new release would definitely be helpful to avoid caching issues.

It needs quite a bit reading between the lines to work out what has happened.

The original incompatibility problem seems to have been included in the innocent-looking Cypress 13.3.0 release on Sep 27, 2023.

At the same time as the 13.3.0 release, and based on users reporting they didn't change anything, whilst workflows started to fail, this appears to be due to Cypress binaries for previously released versions 13.0.0, 13.1.0 and 13.2.0 being replaced.

The remediation was then to replace the whole set of Cypress 13.x binaries again. This breaks the model of Cypress binaries being immutable once released, which is something that caching mechanisms rely on.

[JessefSpecialisterren]

I have retrieved cached binaries of Cypress 13.2.0 from different download dates from our CI machine pool and it does indeed look like there are significant differences between them. Haven't had the time to dive into it, but the timestamps on the files are all different, and some files appear in one and not in the other (including the infamous Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node)

[josephearl]

Can confirm the latest changes have fixed my issue on RHEL8

[MikeMcC399]

Breaking changes were announced in Changes in defense of Cypress Intellectual Property regarding Cypress 13.x. There hasn't been any statement linking this issue to those changes, however it seems the most likely explanation.