[Proposal] Support AL2023/NodeJS18 for OpenSearch/Dashboards Releases

[peterzhuamazon]

With the announcement of Amazon Linux 2023 Preview available for testing, we want to setup some notes here for the support of AL2023 once it is fully released to public.
https://aws.amazon.com/about-aws/whats-new/2021/11/preview-amazon-linux-2022/
https://aws.amazon.com/about-aws/whats-new/2023/03/amazon-linux-2023/

• Evaluate the compatibility of AL2023 when running OpenSearch/Dashboards
  • OpenSearch/Dashboards latest binary can run on AL2023 with the same expected behaviors on AL2
  • Working alongside plugin teams to ensure all the plugins are running as expected
  • Performance testing to ensure there is no performance regression
  • Evaluate the package/dependencies availability on AL2023 default repository
  • Evaluate the size of the image vs contents integrity
• Evaluate AL2023 availability
  • AL2023 available on docker for both x64 and arm64
  • AL2023 available on EC2 for both x64 and arm64 for testing
• Evaluate Build Process
  • What level of changes do we need to support AL2023 with current build process and scripts
  • Can we support both AL2 and AL2023 in the same system
  • Build tool related libs/packages/dependencies availability on AL2023
• Evaluate deprecation plan for older OS
  • When would we deprecate AL2 while supporting AL2023
  • Will this introduce breaking changes to customer
  • Would we support older version of OpenSearch/Dashboards with new OS?

We welcome more discussion on this.

Thanks.

[peterzhuamazon]

Hi @nknize @dblock what do you think about this change?

From our teams perspective it makes sense to upgrade the main docker image from AL2 to AL2023 starting 3.0.0. It also make sense that we use this opportunities to upgrade other OS such as Ubuntu and Rockylinux to their latest versions.

Thanks.

[dblock]

I really don't know enough about this @peterzhuamazon. What are pros/cons?

[peterzhuamazon]

Now, with this issue on the horizon, we need to consider how to proceed with the changes.

• Bump Node.js from 14 to 18 Proposal OpenSearch-Dashboards#3601
• Upcoming Transition to Node.js v18: Plugin Compatibility Test Required Overall Issue OpenSearch-Dashboards#4058

This implies:

• We need to deprecate the ci image that utilizes centos7 or al2.
• Our ci and release docker images need to transition from centos/al2 to al2023.
• Migrate build repo code from using Python 3.7 to 3.9 version #3351
  • The Python upgrade from 3.7 to 3.9 need to happen.
  • AL2 has python 3.7 as the default, but AL2023 already includes 3.9. Attempting to compile Python 3.7 on AL2023 would consume significant time.
  • Python 3.7 support will reach its eol in 1 month.
• If we continue using the same ci image for 1.x / 2.x / 3.x, KNN will crash for certain users with older operating systems (older glibc than the one on al2023), as centos7 represents the best compatibility for KNN, having the oldest glibc version on the OS, that does not support Node18.
  • Use containers for library artifact build opendistro-for-elasticsearch/k-NN#169
  • Unless we create a new jenkinsfile specifically for 1.x, we cannot extensively modify the existing dist-build-dashboards jenkinsfile, as it is already close to the Jenkinsfile max size limit.
  • Introducing a new jenkinsfile solely for 1.x will cause issues due to many hardcoded dir names as well as structures on S3.
  • One possible approach to avoid this issue is to use centos7 for building OS and use al2023 for building OSD, although this approach lacks standardization and is hard to maintain.
• This change need to occur after the completion of the Jenkins Upgrade.
  • [Upgrade jenkins] Switch to java 11 as default java version on agent nodes opensearch-ci#277
• This is a significant breaking change, and it seems that we want to backport it to 2.x due to the deprecation of node14.

[bbarani]

Tagging @vamshin @seanneumann @AMoo-Miki @ananzh @jmazanec15 for visibility.

[seanneumann]

For the Node 18 upgrade, we're targeting 2.8. This is important as Node 14 no longer will receive security updates (as of April 2023). Is there an ask to delay to solve the above?

[peterzhuamazon]

KNN support on the native code compilation:

node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by node)
node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by node)
node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by node)
node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by node)
node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by node)
node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by node)

# CentOS7
ldd (GNU libc) 2.17

# AL2
ldd (GNU libc) 2.26

# CentOS8/RockyLinux8
ldd (GNU libc) 2.28

# AL2023
ldd (GNU libc) 2.34

Seems like if we want to switch from CentOS7 to CentOS8/RockyLinux8, we will need to remove support of CentOS7 and AL2 from our support matrix on our website as well, since CentOS8/RockyLinux8 is the baseline of node18 with glibc 2.28.

• NodeJS 18 revert to building on CentOS 7, RHEL 7, Ubuntu Bionic 18.04, other other LTS distros nodejs/node#43246
  https://opensearch.org/docs/latest/opensearch/install/compatibility/

And if we compile knn on 2.28 then the lib will crash on glibc that is lower than 2.28.

cc: @krisfreedain

[bbarani]

@vamshin @jmazanec15 Can you please provide your inputs as well?

[krisfreedain]

we will need to remove support of CentOS7 and AL2 from our support matrix on our website as well, since CentOS8/RockyLinux8 is the baseline of node18 with glibc 2.28

I would agree @peterzhuamazon - if we test on CentOS8/RockyLinux8 the compatibility matrix needs to be updated to accurately reflect that for the community

[ananzh]

Hi @peterzhuamazon we will bump the node.js to version 18.16.0

[jmazanec15]

@peterzhuamazon want to clarify, you are saying that node 18 requires 2.28, correct? Then doesn't this break compatibility with AL2 and centos 7? So, regardless of k-NN libs, upgrading to Node 18 would require us to remove centos 7 and AL2 from the compatibility matrix.

If this is not the case, we could also elect to compile k-NN libs in separate docker image to maintain compatibility.

[joshuali925]

@peterzhuamazon not sure if you saw this opensearch-project/OpenSearch-Dashboards#3601 (comment)

unlike chromium, it is possible to do a custom build of nodejs on centos 7 to lower glibc dependencies (I also saw musl builds somewhere, which can be another possibility to workaround this issue). I've not looked into performance or other issues might be caused by downgrading glibc, but if OS upgrade is difficult maybe this would be easier?

[anirudha]

Did anyone confirm node 18 with AL2? @seanneumann

We noticed some glibc related compatibility issues. Not sure if it's an isolated incident.

[peterzhuamazon]

@peterzhuamazon not sure if you saw this opensearch-project/OpenSearch-Dashboards#3601 (comment)

unlike chromium, it is possible to do a custom build of nodejs on centos 7 to lower glibc dependencies (I also saw musl builds somewhere, which can be another possibility to workaround this issue). I've not looked into performance or other issues might be caused by downgrading glibc, but if OS upgrade is difficult maybe this would be easier?

That is likely not an option because CentOS7 is going to be out of support in 1 year, and we are also on track to move to al2023 anyway. Manually rebuild nodejs18 would add another dep on our side, which is not going to scale on the long run.

Thanks.

[peterzhuamazon]

The cmake upgrade to 3.23.3 version in #3706 was initially setup as a follow up in the al2023 upgrade/nodejs upgrade, tho now it seems like a 2.9.0 dependencies thus it needs to be updated soon.

Thanks.

[peterzhuamazon]

AL2023/NodeJS18 on 2.10.0 Release:

Required Steps

• Update all AMIs to have NodeJS14/16/18 support ✅
  • This can hold on a bit as the MacOS build is not on OSD and we are converting Windows to containers:
  • [Bug]: Windows CI builds failing to find docker (Update to Run Windows inside docker containers) opensearch-ci#281
• Update all CI docker images to have NodeJS14/16/18 support ✅
  • [NodeJS18 Upgrade] Add opensearch rockylinux ci docker images changes #3836
• Update all Release docker images to use NodeJS18 as the base node version for OSD ✅
  • [NodeJS18 Upgrade] Add new dockerfiles based on AL2023 and fix incorrectly copied yml and entrypoint #3962
  • [NodeJS18 Upgrade] Add AL2023 Docker Images Selection by BuildDocker lib opensearch-build-libraries#302
  • [NodeJS18 Upgrade] Remove old AL2 Docker Host and Tweak Windows Startup Scripts opensearch-ci#342
  • [NodeJS18 Upgrade] [Windows Container] Adjust spare instance number for AL2023 and Windows Docker Host opensearch-ci#345
• Switch OSD core to have 18.16.0 version ✅
  • [NodeJS18 Upgrade] Bump OpenSearch-Dashboards 2.10.0 to use nodejs 18.16.0 version and update changelog of 2.9.0/Unreleased Sections OpenSearch-Dashboards#4948
• Deprecate all CentOS7 images in build systems and replace with the RockyLinux8 images (Except client release image as it does not require nodejs18 by default for now) ✅
  • [NodeJS18 Upgrade] Switch to AL2023 Runner and Rockylinux8 for NodeJS18 Upgrade #3967
  • [NodeJS18 Upgrade] [Windows Container] Adjust spare instance number for AL2023 and Windows Docker Host opensearch-ci#345
• Work with OSD team members to ensure all OSD core and plugins are locking to NodeJS 18.16.0 at least on 2.x/2.10 branches. 1.x can pending.
  • To my understanding all scripts are using the nvmrc file in OSD for nodejs version in Jenkins build.
• Since KNN already update the dependencies such as gcc to v7 and cmake to 3.23.3, it is easier for us to bring them onto either Rockylinux8 or AL2023. Starting from now k-NN will not be able to run on OS with glibc version lower than 2.28 (previously 2.17)
  • [AUTOCUT] Distribution Build Failed for k-NN-2.9.0 (Unittest error on LINUX and FAISS compilation on Windows) k-NN#975 (comment)
  • Getting Segfault while compiling the Faiss (faiss::{anonymous}::kernel_accumulate_block) on ARM facebookresearch/faiss#2955
• Work with doc team to update the doc compatibility charts to remove CentOS7 support and add AL2023 ✅
  • [NodeJS18 Upgrade] Update support matrix for node18 documentation-website#4303
    • [WIP] Update compatibility chart for OS support related changes documentation-website#4980
  • 20230912: We have decided to restore the b/c of k-NN on 2.10.0 and will revisit the deprecation of certain OS later
• Test all the pipelines on the new servers.

Optional Steps

• Need to discuss after the 2.10.0 release whether or not 1.x line need to upgrade as well.
  • 1.x will keep using the old centos7 docker images as I have set measures to keep it that way
  • 20230912: We will keep 2.x on CentOS7 for OpenSearch on to b/c k-NN before fully deprecating old OSes with glibc<2.27/2.28 later
    • Switch OpenSearch Build Image Back to CentOS7 #4002
• Have separate pipeline to test NodeJS14 bc abilities.
  • No need, one pipeline is enough for that.

• Updating

[wbeckler]

What is the status of the separate pipeline to test node 14 backward compatibility? There are going to be users using old linux versions who can't go past node 14 yet, and it would be useful to know what fixes we can do so as to not unnecessarily break things for those users.

[bbarani]

@wbeckler We do not have plans to create a separate pipeline in public Jenkins infra for EOL versions of Node. If needed, this needs to be tackled at repo level.

[peterzhuamazon]

Have quick talk with OSD @AMoo-Miki about core, and we are planning to switch 2.x to 18.16.0 next week to start testing on the 2.10.0 build with newer nodejs.

Thanks.

[peterzhuamazon]

The majority of the tasks in this issue is resolved and implemented.
We will issue new tickets whenever new tasks coming up.

Thanks.