NodeJS 18 revert to building on CentOS 7, RHEL 7, Ubuntu Bionic 18.04, other other LTS distros

[theofficialgman]

What is the problem this feature will solve?

Moving to RHEL 8 has raised the glibc version being linked against
(2.28). The official Node.js Linux release binaries will only run on
Linux distributions with a matching or higher version of glibc.

#42659
nodejs/build#2815
nodejs/build#2741

What is the feature you are proposing to solve the problem?

Building on one of these older distros. All of them have support well past the EOL of NodeJS 18 when paired with paid or free (through sponsorship) extended support.

The breaking change that has been implemented will cause many projects dependent on NodeJS to drop centos 7, rhel 7, and (more importantly) Ubuntu bionic and Ubuntu 18 (bionic in snaps).

One of the most important projects dependent on nodeJS 18 using bionic or an older distro is Electron https://www.electronjs.org/docs/latest/faq#when-will-electron-upgrade-to-latest-nodejs This of course will trickle down to 1000s of dependent projects. Many developers use the official nodejs binaries in their projects

if we compare the linux support to window support, we can clearly see that linux is the least backwards supported platform requiring glibc 2.28 which released in August 1st 2018 (and wasn't included in distros until even later), while Windows has support for versions going back to 2015 for tier 1 support.

you say:

we do not want to have to make a disruptive change
towards the end of Node.js 18's life cycle.

when this would actually be the best thing to do. support the LTS versions for as long as the LTS version support exists. You can of course support it further than the LTS distros are supported (by purchasing or using the sponsored extended support).

What alternatives have you considered?

Notify all open source projects on github to NOT use NodeJS 18 through a custom github bot and to remain on prior NodeJS releases (eg, 16.x) until absolutely necessary.

[cobalt2727]

This would also solve nodesource/distributions#1392 - right now, the NodeSource build system is shipping completely broken packages to users, and has been for over a month now, with no official response given as to how this will be addressed.

[rapito]

Albeit upgrading some linux distros to newer versions because of security updates and such would practically "get rid of the issue", some CI tools (in my case running on Ubuntu 18.04) depend on some scripting pipelines made in nodejs. It's dependency of glibc is breaking builds and automated pipelines on many environments for me.

Notably this is not addressing any real security patches or updates that would justify forcing glibc on the OS.

[targos]

@nodejs/build

[bnoordhuis]

One good reason for dropping support for CentOS/RHEL 7 is glibc. CentOS 7 ships with glibc 2.17 but node's dependencies are moving to newer versions.

(In fact that already happened but we're working around it for now. Not a viable long-term strategy though.)

Ubuntu 18.04 ships with glibc 2.27. That's probably new enough to last until v18's EOL.

[rvagg]

Some background can be found in nodejs/build#2815, and specifically on this @ nodejs/build#2741

But the headline is this:

• "CentOS-7 updates until June 30, 2024"
• Node.js 18.x EOL on 2025-04-30

Unfortunately this happens on a regular cadence, we have to deal with this every few years, as do other major projects. It sucks for users, I hate it as someone who maintains a few systems on ancient versions of Linux. But you have the option of building Node.js yourself - if you can - V8 is going to get in your way over time here as it moves faster in its minimum requirements than we probably would be and we've regularly been forced to up our minimums just to support V8 (sometimes we manually patch to maintain backward compatibility!).

[cobalt2727]

nodejs/node-v8#220
V8 requires 2.27, no? What other dependencies, if any, are getting in the way?
nodejs/build#2815 (comment)
And if you guys have ESM licenses, all the more reason to continue to target 18.04.

[itmustbe]

I also wish that there had at least been a warning that Node 18.x.x was incompatible with CentOS 7. And I'd certainly cast a vote for maintaining compatibility with CentOS 7 and similar distros if possible!

Although I'm only a tinkerer with Linux, I do have a significant non-profit website running on my little server. I try to keep everything up-to-date on my server, and during a routine update of Node.js, I encountered the dreaded 503 error on the website I'd just updated (thankfully a staging site).

I've rolled back to 17.9.0 for the foreseeable future, but it always makes me nervous having things like Node.js getting more and more out-of-date.

CentOS 8 is something I'm planning on, but it requires planning and time to migrate, and I will not have the time to dedicate to that project for awhile yet. I also lack the knowledge on building Node.js myself.

[mhdawson]

There was a warning in the release blog post - https://nodejs.org/en/blog/announcements/v18-release-announce/#toolchain-and-compiler-upgrades.

@itmustbe for the future what do you think would have been the best channels to use in advance of the release itself to get the news out?

[cobalt2727]

Once again, V8 is a non-issue for 18.04. They're explicitly still supporting 18.04 by making the target glibc 2.27, which 18.04 runs with.

[mhdawson]

@cobalt2727 I think the comment above in #43246 (comment) sums it up well. There are a number of different factors that went into the decision available (build team resources, support time lines, deps, future risk, etc.) and with a supported version of Node.js 16.x being available for another 2 years it's the balance we chose between the needs of the project and ends users.

[cobalt2727]

I would like to invite you guys to reconsider and reread the original post - breaking support for an OS that's perfectly capable of running NodeJS has a high chance of causing problems in thousands of projects for anyone on 18.04 for whatever reason well before before EoL hits. The balance you're describing only works if projects just... don't use 18 for a while, and lose out on feature updates until then.

So far, other than a misunderstanding of V8 requirements, I don't see the issue with continuing to provide official builds for 18.04. May as well put those ESM licenses to good use, yeah?

[richardlau]

We don't have ESM licenses. nodejs/build#2815 (comment) mentions that they were not followed up on.

[itmustbe]

I appreciate there are many aspects to Node.js development that I don't closely follow, and probably should have.

What happened in my case is that I run a virtual private server on CentOS 7 with Virtualmin/Webmin, and an update was flagged "Version 18.2.0 is available". Nothing stopped me from pushing the button to update Node.js on an unsupported system, and I rather wish something had done, as the website in question immediately went down with a 503 error for now-obvious reasons (not supporting the latest GLIBC etc. owing to the outdated version of libstdc++).

I contacted the folks who manage Virtualmin/Webmin, and they will be locking old distros to the correct version of Node.js so that this doesn't keep happening to other people in similar situations.

[mhdawson]

@itmustbe thanks for the update/response. It does seem that stopping the notice that an update is available when it is not would help.

[cobalt2727]

We don't have ESM licenses. nodejs/build#2815 (comment) mentions that they were not followed up on.

My mistake, sorry about that. Nonetheless, I think it's still worth supporting just for one more year.

[theofficialgman]

electron and chromium have devised a very smart and interesting method for continuing to support GLIBC 2.17+ on their own builds for related projects. instead of using outdated distros to build on, they use distros like bullseye and sid but implement a binary rewrite script to relink the compiled binaries against glibc 2.17 and older functions.
https://source.chromium.org/chromium/chromium/src/+/main:build/linux/sysroot_scripts/reversion_glibc.py

this should be investigated for inclusion in NodeJS builds

[theofficialgman]

accompanying this script, they also make patches to the sysroot they use to build on so that it will build binaries that can be successfully linked on glibc 2.17+
a few important lines from it are here: https://chromium.googlesource.com/chromium/src/+/HEAD/build/linux/sysroot_scripts/sysroot-creator.sh#324

[bnoordhuis]

I'm aware of that technique (another approach is to alias the symbols in asm) but it's hacky and fragile.

I can see it working for a project like chromium where there's a team of full-time engineers working to keep the build in good shape. For an OSS project that wants to maintain a semblance of stability it's a much riskier proposition.

You can of course run that script locally and cross your fingers.

[theofficialgman]

might I ask then, what is the point of switching to a fully maintained distro anyway? you are using it to compile code, you only need functional GCC/G++. Each of the distros already mentioned in the title of this post are perfectly suitable for running modern compiler toolchains on.

yes I have fully read all of the relevant PRs, Issues, discussions, etc and this has never been publicly discussed. It appears to me that you just wish to update for the purpose of "gaining support" when that support doesn't actually do you any good. these are build servers. GCC packages haven't changed in years (for the specific version you target)

[bnoordhuis]

You yourself brought up that glibc symbol hack so you must be aware that it involves more than just a "functional GCC/G++".

[theofficialgman]

You yourself brought up that glibc symbol hack so you must be aware that it involves more than just a "functional GCC/G++".

the symbol hack was so that you could continue to build on newer distros with higher version of glibc and then patch the versions of the symbols to be lower than that of the glibc shipped in the distro. there isn't anything that is in V8 that won't compile already with glibc 2.27 (or older, chromium still targets that).

obviously it would be easier to simply use an older distro which already has that older glibc version, but I was just stating options

[theofficialgman]

We don't have ESM licenses. nodejs/build#2815 (comment) mentions that they were not followed up on.

this would probably be the best avenue to pursue. you need to keep in mind that the change from glibc 2.17 (what previously was required) to glibc 2.28 (what is now) is a full 6 YEAR version bump. that is huge

we also already know the glibc 2.27 bump was accidental and reverted: v8/v8@4e81f25 . they do not, and have not, done anything that requires glibc 2.27, 2.28, or anything modern. chromium and v8 still intend and have been building with a minimum of 2.17 in mind. Any attempt to say differently is misguided.

[bnoordhuis]

I don't think you can construe the conversation in https://bugs.chromium.org/p/v8/issues/detail?id=12682 as the V8 team saying they plan on supporting glibc 2.17 indefinitely. Even if they did, that's just one of Node's dependencies.

from glibc 2.17 (what previously was required) to glibc 2.28 (what is now) is a full 6 YEAR version bump

Sure, but that's because 2.17 is just really old. It was released in 2012. It's 2022 now!

[cobalt2727]

I don't think you can construe the conversation in https://bugs.chromium.org/p/v8/issues/detail?id=12682 as the V8 team saying they plan on supporting glibc 2.17 indefinitely.

The fact that it was deemed a critical enough bug to get fixed in 10 days is more than enough to show they intend to keep supporting legacy glibc versions. So when deciding whether or not to upgrade the build system for NodeJS, v8 is evidently not something to take into consideration, period.

Even if they did, that's just one of Node's dependencies.

Now that we've established v8 is a non-issue, what other dependencies are getting in the way?

[bnoordhuis]

Not going to engage. I don't get the feeling you're arguing in good faith. At the very least your tone is off-putting.

[cobalt2727]

Then I'd like to apologize for that - but I'm honestly wondering what other roadblocks there are.

[sbwml]

Node.js latest version for CentOS 7 (glibc 2.17) https://github.com/sbwml/node-latest-centos

[theofficialgman]

@richardlau please attempt the following oneliners on your x64 and arm64 buildservers respectively to prevent linking to functions when not necessary. as long as building nodejs does not consume a library that already links to glibc2.28 then this should lower the requirements back to 2.17. simply execute the sed commands and then build nodejs from scratch.

x64 or arm64 native compilation

# __GLIBC_MINOR__ is used as a feature test macro.  Replace it with the
# earliest supported version of glibc 2.17 as was previously the case
sudo sed -i 's|\(#define\s\+__GLIBC_MINOR__\)|\1 17 //|' "/usr/include/features.h"
# fcntl64() was introduced in glibc 2.28.  Make sure to use fcntl() instead.
sudo sed -i '{N; s/#ifndef __USE_FILE_OFFSET64\(\nextern int fcntl\)/#if 1\1/}' "/usr/include/fcntl.h"

arm64 cross compiling

# __GLIBC_MINOR__ is used as a feature test macro.  Replace it with the
# earliest supported version of glibc 2.17 as was previously the case
sudo sed -i 's|\(#define\s\+__GLIBC_MINOR__\)|\1 17 //|' "/usr/aarch64-linux-gnu/include/features.h"
# fcntl64() was introduced in glibc 2.28.  Make sure to use fcntl() instead.
sudo sed -i '{N; s/#ifndef __USE_FILE_OFFSET64\(\nextern int fcntl\)/#if 1\1/}' "/usr/aarch64-linux-gnu/include/fcntl.h"

also please double check that those filepaths are the same on centos. these are ubuntu's standard paths.

this will have no affect on the libstdc++ version that nodejs is dependent on however it is already low enough to be compatible with ubuntu bionic on x64/arm64 libstdc++ >= 6.0.25 (GLIBCXX_3.4.25).

[theofficialgman]

@richardlau see above message please.
Unfortunately I can't help you out any more than this since your build scripts and system images/containers are all closed source 😔

[luke-hill]

I was redirected here by another conversation. Not sure if this is the appropriate forum so apologies if not.

Given that ubuntu 18 is very much a supported version and node is meant to be supporting supported software, I would say this should be a priority. Failing that, there should be either some restrictions / build failures that indicate this software is no longer supported, alongside some form of article/press release stating that node won't support ubuntu 18 and below.

Migrating a piece of software mid-cadence from supported to unsupported with little to no documentation is why you're seeing huge threads popping up with lots of discussion. Node identifies with following semver, so ideally it needs to pick a camp.

Obviously it's well within the rights of maintainers to support whatever versions they so desire. But if you're going to change things, please articulate why so people can follow / adapt / fix. This is part of the reason I'm a lot more apprehensive with JS packages compared to other languages because of the propensity for these large changes happening with little-no notice.

[theofficialgman]

@richardlau @rvagg please give this a shot. takes 2 mintues of your time. should lower the requirements to 2.17 (assumes cross compilers in standard ubuntu paths and deb commands, you should be able to pretty easily adapt for RHEL with rpm via similar methods)
libgdx/Jamepad@5eb5f30

[theofficialgman]

I'd implement that for you but unfortunately your buildscripts and server configs are still closed source nodesource/distributions#1491

[theofficialgman]

Looks like official NodeJS builds do NOT work on Debian buster either
https://nodejs.org/dist/v20.3.1/node-v20.3.1-linux-armv7l.tar.xz
node: /lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by node)

[theofficialgman]

@richardlau docs are incorrect, see above comment. NodeJS binary builds not compatible with GLIBC 2.28 on armv7l
https://github.com/nodejs/node/blob/main/BUILDING.md#official-binary-platforms-and-toolchains

[leqii-com]

我想问下大佬们，node18及以上版本在centos7和8上都不被支持了吗？因为今天centos7.5系统上运行node18.0.0，安装后用不了，报了如下错误：
node: /lib64/libm.so.6: version GLIBC_2.27' not found (required by node) node: /lib64/libc.so.6: version GLIBC_2.25' not found (required by node)
node: /lib64/libc.so.6: version GLIBC_2.28' not found (required by node) /usr/local/bin/node: /lib64/libm.so.6: version GLIBC_2.27' not found (required by /usr/local/bin/node)
/usr/local/bin/node: /lib64/libc.so.6: version GLIBC_2.25' not found (required by /usr/local/bin/node) /usr/local/bin/node: /lib64/libc.so.6: version GLIBC_2.28' not found (required by /usr/local/bin/node)

[kajtzu]

我想问下大佬们，node18及以上版本在centos7和8上都不被支持了吗？因为今天centos7.5系统上运行node18.0.0，安装后用不了，报了如下错误：

You can find the URL to unofficial builds here nodejs/unofficial-builds#69 (comment), they will work on CentOS7. You will want an archive ending glibc217.tar.gz or .xz.

HTH.

[leqii-com]

@kajtzu 非常感谢你，真的可以，根据你给的帖子，下载了这个https://github.com/momiji/nodejs-unofficial-builds/releases/download/v1.0.0/node-v18.14.2-linux-x64-glibc-217.tar.gz
哈哈！Thanks very much

[BlackSunshine-manage]

@kajtzu 非常感谢你，真的可以，根据你给的帖子，下载了这个

Thank you very much, I pray for your health

[OneCodeMonkey]

真是坑啊，你们不知道 Centos7 的市场保有量多大吗？说不支持 就不支持了？？

[kajtzu]

What a pit, don't you know how much Centos7 has in the market?

It is EOL 30.6.2024 just like RHEL7 ;)

[sbwml]

Announcing up to 4 years of Extended Life Cycle Support (ELS) for Red Hat Enterprise Linux 7

https://www.redhat.com/en/blog/announcing-4-years-extended-life-cycle-support-els-red-hat-enterprise-linux-7

[leqii-com]

@OneCodeMonkey 没办法，谁叫咱要用他们的东西，所以要摆脱依赖

[javaaiorg]

Use Node.js no GCC depends version can solved this problem:
https://www.javaai.org/a/c892d1e98fd94a64018ff4a88ec9000c

[leqii-com]

@javaaiorg Get。太牛了，没有GCC也能用nodered，长见识了，十分感谢。

[mhdawson]

@sbwml it's necessary to look carefully about what is included/not included in the Extended Life Cycle Support (ELS) for Red Hat Enterprise Linux 7. I'm not the authority but I don't believe it includes all runtimes.