yarn version incompatibility issue blocks Dashboards plugins CI, docker-image, windows build

[zhongnansu]

❗This issue may block release 2.6

issue description

Multiple Dashboards plugins CI failure reported during plugin bootstrap step, after this commit to Dashboards core merged recently. @opensearch-project/opensearch@2.1.0: The engine "yarn" is incompatible with this module. Expected version "^1.22.10". Got "1.21.1"

• dashboards-observability plugin CI
• dsahboards-maps plugin CI

yarn run v1.21.1
$ node ../../scripts/osd bootstrap
 info [opensearch-dashboards] running yarn

$ node ./preinstall_check
[1/5] Validating package.json...
[2/5] Resolving packages...
warning Resolution field "typescript@4.0.2" is incompatible with requested version "typescript@~4.5.2". 
[3/5] Fetching packages...
warning sha.js@2.4.11: Invalid bin entry for "sha.js" (in "sha.js").
info fsevents@2.3.2: The platform "linux" is incompatible with this module.
info "fsevents@2.3.2" is an optional dependency and failed compatibility check. Excluding it from installation.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
error @opensearch-project/opensearch@2.1.0: The engine "yarn" is incompatible with this module. Expected version "^1.22.10". Got "1.21.1"
error Found incompatible module.
ERROR [bootstrap] failed:
ERROR Error: Command failed with exit code 1: /opt/hostedtoolcache/node/14.[20](https://github.com/opensearch-project/dashboards-maps/actions/runs/4206519945/jobs/7300112650#step:13:21).1/x64/lib/node_modules/yarn/bin/yarn.js install --non-interactive
          at makeError (/home/runner/work/dashboards-maps/dashboards-maps/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:25150:11)
          at handlePromise (/home/runner/work/dashboards-maps/dashboards-maps/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:24085:26)
          at processTicksAndRejections (internal/process/task_queues.js:95:5)
          at async installInDir (/home/runner/work/dashboards-maps/dashboards-maps/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:238[21](https://github.com/opensearch-project/dashboards-maps/actions/runs/4206519945/jobs/7300112650#step:13:22):3)
          at async Project.installDependencies (/home/runner/work/dashboards-maps/dashboards-maps/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:14780:5)
          at async Object.run (/home/runner/work/dashboards-maps/dashboards-maps/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:9003:11)
          at async runCommand (/home/runner/work/dashboards-maps/dashboards-maps/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:57870:5)
          at async Module.run (/home/runner/work/dashboards-maps/dashboards-maps/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:[26](https://github.com/opensearch-project/dashboards-maps/actions/runs/4206519945/jobs/7300112650#step:13:27)3:3)
error Command failed with exit code 1.

Root cause

The recent commit in OpenSearch Dashboards updates dependency @opensearch-project/opensearch from ^1.1.0to ^2.1.0, where openearch-js2.1.0 specifies yarn: ^1.22.1, in this commit.

Plugin CI workflow fetches yarn version from Openesarch-Dashbaords/package.json file, to set up yarn, and still found 1.21.1. This is causing the yarn version incompatibility, and bootstrap will fail.

Dashboards docker image, windows build scripts are using hardcoded yarn version, which will potentially be impacted, according to @peterzhuamazon

Impact

• All Opensearhc Dashboards Plugin CI & build
• Dashboards docker image build, windows build process, etc

Proposed solution

• step 1: To unblock dashboards plugins CI, Opensearch-Dashboards core repo needs to update yarn version in package.json to ^1.22.1 Upgrade yarn version to be compatible with @openearch-project/opensearch OpenSearch-Dashboards#3443
• step 2: After step 1 is completed, to unblock docker image build, windows build, we could fetch the yarn version on demand from Openesarch-Dashbaords/package.json , instead of hardcode.

[peterzhuamazon]

This issue is brought up late as plugins are not always following the core defined yarn version.

After quick glance around 5 docker images and 1 ami get impacted.

./release.centos.clients.x64.arm64.dockerfile:    for node_version in $NODE_VERSION_LIST; do nvm install $node_version; npm install -g yarn@^1.21.1; done
./build.centos7.opensearch-dashboards.x64.arm64.dockerfile:    for node_version in $NODE_VERSION_LIST; do nvm install $node_version; npm install -g yarn@^1.21.1; done
./test.rockylinux8.systemd-base.x64.arm64.dockerfile:RUN npm install -g yarn@1.22.18
./test.rockylinux8.opensearch-dashboards.x64.arm64.dockerfile:RUN npm install -g yarn@1.22.18
./test.centos7.performance-test.x64.dockerfile:    for node_version in $NODE_VERSION_LIST; do nvm install $node_version; npm install -g yarn@^1.21.1; done

./scripts/windows/scoop-install-commons.ps1:volta install yarn@^1.21.1

Among these 6 there are 2 docker images related to bundle build + integTest, and 1 ami related to bundle build, means they need to be updated now to avoid failures on 2.6.0 builds.

The rest of them can be updated at a later time.

The updates of these images/ami should retrieve the version of yarn dynamically from core osd repo, instead of hardcoding again in the template.

Thanks.

[peterzhuamazon]

Since this issue would impact on 2.6.0 release, cc @CEHENKLE @bbarani into the discussion on this.

Thanks.

[kristenTian]

Would it be easier to mitigate by downgrading the yarn version in openearch-js to be compatible with opensearch-dashboard?

[peterzhuamazon]

PRs:

• Add RPM systemd images with non-root sudo support #3191 (image updated)
• Add Yarn version updates to the ci images #3212 (pending update)
• Add yarn version update scripts on Windows opensearch-ci#253 (image updated, require deployment)

[joshuarrrr]

Why don't we just revert opensearch-project/OpenSearch-Dashboards@de89c52 in 2.x?

In general, OpenSearch Dashboards considers bumping direct dependencies (such as @opensearch-project/opensearch) to be a breaking change when the dependency bump contains a breaking change (which it clearly does in this case).

[joshuarrrr]

Revert PR: opensearch-project/OpenSearch-Dashboards#3464

[kavilla]

+1 @joshuarrrr. I think the call if we want the sigv4 into OpenSearch Dashboards core, without a deep dive, I don't see why not cut a minor release OpenSearch JS client for 1.x to have this feature. And then have OpenSearch Dashboards take the minor version bump from the client. However, I will note that client does not appear to be following the branching strategy of the project while also not deprecating previous versions. If I'm just a regular NPM user, I would expect that OpenSearch JS 1.x is being supported even though it is not actually being supported.

cc: @dblock @CEHENKLE @bbarani

Granted I only checked one other repo's 1.x branches, it would appear that clients aren't actively supporting 1.x versions. Is this intentional and called out somewhere?

[peterzhuamazon]

Adding @zhongnansu @derek-ho on to the change on this.
Do we need to make the changes to make the revert.

Thanks.

[zhongnansu]

@joshuarrrr
I am against reverting opensearch-project/OpenSearch-Dashboards@de89c52 only because it includes opensearch-js version bump

• openearch-js client 2.0 was released June, 2022. But in OSD core, we are still using 1.xclients now. According to the compatibility matrix of opensearch-js client. Client version 1.x doesn't support OpenSearch version 2.x, which means OSD was supposed to bump opensearch-js client lib.

• As for dashboards plugins, in any 2.x.x.x, they know they are talking to OpenSearch 2.x. But opensearch-js client1.x is being used, as it's decided from upstream OSD core. But till now, there's no known issue complaining that plugin failed to talk to OpenSearch 2.x using opensearch-js client 1.x, which means plugins are not using any deprecated APIs. So there shouldn't be a problem with plugin consuming opensearch-js 2.x moving forward

• Plus, this commit is a feature commit that was put on the roadmap, and targeted to release in 2.6.0. Also, no backwards compatibility CI has ever failed in any PR during feature implementation stage.

cc: @zengyan-amazon @peterzhu1992 @bbarani

[kavilla]

Client version 1.x doesn't support OpenSearch version 2.x, which means OSD was supposed to bump opensearch-js client lib.

Might be a confusing matrix but on our site we state:

All clients are compatible with any version of OpenSearch.

[SOURCE]

Plus, this commit is a feature commit that was put on the roadmap, and targeted to release in 2.6.0. Also, no backwards compatibility CI has ever failed in any PR during feature implementation stage.

For build wise we usually rely on the build CI to verify and have reverted commits in the past.

[zhongnansu]

For build wise we usually rely on the build CI to verify and have reverted commits in the past.

@kavilla @joshuarrrr
That's why I tried to backport the fix to OSD 2.x, so that plugin CI on 2.x could be triggered to verify. opensearch-project/OpenSearch-Dashboards#3465.
The original plugin CI failure reported in this issue has been addressed in main. https://github.com/opensearch-project/dashboards-maps/actions/runs/4227411858/jobs/7341865358

[kristenTian]

Could we at least test if the yarn version bump would fix the build?

For build wise we usually rely on the build CI to verify and have reverted commits in the past.

@kavilla @joshuarrrr That's why I tried to backport the fix to OSD 2.x, so that plugin CI on 2.x could be triggered to verify. opensearch-project/OpenSearch-Dashboards#3465. The original plugin CI failure reported in this issue has been addressed in main. https://github.com/opensearch-project/dashboards-maps/actions/runs/4227411858/jobs/7341865358

[peterzhuamazon]

Awaiting:

• [Manual backport 2.x][MD] Support SigV4 as a new auth type of datasource (#3058) OpenSearch-Dashboards#3477

[zhongnansu]

@peterzhuamazon
Plugin CI on main is unblocked. But the fix needs to be backported to OSD 2.x, otherwise the same issue will happen.
There's a backport PR opened but blocked: opensearch-project/OpenSearch-Dashboards#3465

[zhongnansu]

Awaiting:

• [Manual backport 2.x][MD] Support SigV4 as a new auth type of datasource (#3058) OpenSearch-Dashboards#3477

@peterzhuamazon This PR is merged to 2.x, but I start to see Plugin CI fail on 2.x, because of same yarn incompatibility. https://github.com/opensearch-project/dashboards-observability/actions/runs/4236740246/jobs/7361893066 , and the PR to fix it from OSD side is blocked. opensearch-project/OpenSearch-Dashboards#3465

Adding @AMoo-Miki, since I saw you opened a related issue in plugin repo about the yarn config in CI workflow. I do want to mention that the same way to configure yarn is used across almost all dashboards plugins in Opensearch Org. opensearch-project/dashboards-observability#290

[bbarani]

Tagging @gaiksaya as she is the release manager for 2.6.0 release.

[gaiksaya]

Hi @zhongnansu @peterzhuamazon What are the next steps here? As the today is the code cutoff date for 2.6.0 trying to see if we need to extend that from dashboards prespective.
Thanks!

[peterzhuamazon]

All images updated now in docker hub (3131 3126 3125 3123 docker-build Jenkins).
Windows image requires a redeployment later.
Thanks.

[gaiksaya]

@peterzhuamazon I do not see any PRs for adding yarn version https://github.com/opensearch-project/opensearch-build/tree/main/docker/ci/dockerfiles/current am I missing anything?

[zhongnansu]

Hi @zhongnansu @peterzhuamazon What are the next steps here? As the today is the code cutoff date for 2.6.0 trying to see if we need to extend that from dashboards prespective. Thanks!

The next step is approve and merge in this PR: opensearch-project/OpenSearch-Dashboards#3465, to fix yarn issue and unblock plugin CI @gaiksaya

[AMoo-Miki]

All the effected plugins were using a flawed logic to install Node.js and Yarn runtimes based on the information they were reading from OSD's manifest. They all took the ranges OSD provided, manipulated it incorrectly and attempted to install Node and Yarn. I have gone through all the repos and raised PRs with the correct procedure on all of them.

There is also a possibility of some repos using strict versions for their own engines.* which is a bad and wrong practice. While I did not see this in any repo, i caught one PR that attempted to do so and commented on it.

There is also a repo that uses a file named .node-version that we are investigating to remove in the future; I will reach out to them if we decide to remove the file so that they can use .nvmrc instead.

[AMoo-Miki]

The next step is approve and merge in this PR: opensearch-project/OpenSearch-Dashboards#3465, to fix yarn issue and unblock plugin CI @gaiksaya

That PR is not an acceptable solution; OSD cannot make arbitrary changes to accommodate mistakes in CI scripts of plugins.

[seraphjiang]

@AMoo-Miki
We should have centralized solution for CI (Build and Test)
It will be a surprise if we use two different yarn to build OSD and Plugin

@peterzhuamazon @bbarani @dblock @seanneumann

[kristenTian]

Should we also add the .npmrc to enforce the restriction the node version in OSD core?

[AMoo-Miki]

Kristen, my goal is relaxing all of these restrictions. If nvmrc would allow me, I would have used a range there as well.

[peterzhuamazon]

All new images for Jenkins have been deployed with yarn 1.22.19.