chore: fix flaky smoke tests on yarn by AlbinaBlazhko17 · Pull Request #2764 · Redocly/redocly-cli

AlbinaBlazhko17 · 2026-04-22T12:14:51Z

What/Why/How?

I found the root cause of the failing tests. Old redoc dependency has a v0.22.0 version of @redocly/config and yarn tries to resolve this version instead of latest.

The root cause: Yarn v1's flat-hoisting is non-deterministic when the install graph contains two incompatible versions of the same package.

Changes:

Migrated to yarn install and use yarn add only for redoc. Because yarn add doesn't respect resolutions property in package.json, which overrides deps.
Added *.tgz file in package.json.
Added pin-config-version.sh script to read version of @redocly/config and @redocly/ajv from openapi-core and write proper version to package.json for smoke tests.
Added to steps with redoc one bash script to inject it to the package.json.
Added to tests missing workaround for cache issue on windows. Fixed existing workaround: removed try catch, because bash exiting non-zero doesn't throw — it sets $LASTEXITCODE. The old catch block would never fire, so on the final attempt the step would exit 0 and the job would be reported.

After changes we have only tiny warning, which means, that the version now resolves properly.
warning Resolution field "@redocly/config@0.48.1" is incompatible with requested version "@redocly/config@0.22.0"

Reference

Resolves #2618

Testing

Ran 10 times smoke tests on this PR

Screenshots (optional)

When test failed:

When test passed:

Check yourself

This PR follows the contributing guide
All new/updated code is covered by tests
Core code changed? - Tested with other Redocly products (internal contributions only)
New package installed? - Tested in different environments (browser/node)
Documentation update has been considered

Security

The security impact of the change has been considered
Code follows company security practices and guidelines

Note

Low Risk
Low risk: changes are limited to CI smoke-test setup and dependency installation behavior, with no production code impact. Main risk is unintended CI breakage due to new install/pinning steps across npm/yarn/pnpm and Windows runners.

Overview
Refactors the smoke-test GitHub Actions workflow to stop installing the CLI tarball via add and instead run npm/yarn/pnpm install, with a dedicated step to inject redoc only for the --redoc matrix jobs.

Adds a smoke-test package.json dependency on @redocly/cli via file:./redocly-cli.tgz, plus scripts (pin-intersecting-deps.sh, add-redoc.sh) to pin @redocly/config/@redocly/ajv via Yarn resolutions and to add redoc@latest to avoid non-deterministic Yarn v1 hoisting.

Improves Windows smoke jobs by clearing npm cache and retrying installs correctly (removing ineffective try/catch and exiting with the final failure code).

^{Reviewed by Cursor Bugbot for commit 92985f2. Bugbot is set up for automated code reviews on this repo. Configure here.}

changeset-bot · 2026-04-22T12:14:56Z

⚠️ No Changeset found

Latest commit: d39f863

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

github-actions · 2026-04-22T12:17:05Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	80.04% (🎯 80%)	7100 / 8870
🔵	Statements	79.45% (🎯 79%)	7379 / 9287
🔵	Functions	83.45% (🎯 83%)	1428 / 1711
🔵	Branches	71.5% (🎯 71%)	4799 / 6711

File Coverage

No changed files found.

Generated in workflow #9617 for commit 92985f2 by the Vitest Coverage Report Action

github-actions · 2026-04-22T12:17:11Z

CLI Version	Mean Time ± Std Dev (s)	Relative Performance (Lower is Faster)
cli-latest	3.099s ± 0.034s	▓ 1.01x
cli-next	3.082s ± 0.021s	▓ 1.00x (Fastest)

tatomyr · 2026-04-23T06:30:05Z

        with:
          node-version: 24
-      - run: bash ./tests/smoke/basic/run-smoke.sh "npm i redoc redocly-cli.tgz" "npm run"
+      - run: bash ./tests/smoke/basic/run-smoke.sh "npm install && npm i redoc" "npm run"


This doesn't make any difference but the intention becomes less clear.

I wanted to use yarn install instead of yarn add to fix the issue, so i added into package.json dependency, which points to the tgz cli file. I changed all tests to use just install and be consistent across the tests.

tatomyr · 2026-04-23T06:31:36Z

      - run: |
          for i in {1..2}; do # workaround for yarn cache issue
-            sleep 5 && bash ./tests/smoke/basic/run-smoke.sh "yarn add ./redocly-cli.tgz" "yarn" && break
+            sleep 5 && bash ./tests/smoke/basic/run-smoke.sh "yarn install --no-lockfile" "yarn" && break


What does this change? There's no lockfile before the install, and we're installing the dependencies only once during this job, so having or not having the lockfile should change nothing.

You are right. Let me drop this arg and re-ran tests, because my concern is that we are running yarn install and then yarn add redoc and the second command may overwrite the lock file and @redocly/config as a result.

tatomyr · 2026-04-23T06:33:29Z

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+REDOCLY_CONFIG_VERSION=$(node -e "const fs = require('fs'); const p = JSON.parse(fs.readFileSync('./packages/core/package.json', 'utf8')); console.log(p.dependencies['@redocly/config'].replace(/\^/, ''));")


We're emulating something that will not be the case in our users' setup, so the smokes become irrelevant. Let's try keeping them as much close to the reality as possible.

This bash script read the version of @redocly/config from openapi-core and write this version to package.json of smoke tests to override this dep to be sure, that all tests will resolve proper version of @redocly/config. Users will resolve the same version of @redocly/config. My first try to fix this issue is manually adding this properties to the package.json, but we can forget to change it there, so i decided to create a script.

Let's discuss this offine.

tatomyr · 2026-04-23T06:34:34Z

-# Executing the command provided as the first argument 
-$1
+# Executing the command provided as the first argument
+eval "$1"


What's the difference?

Before this change, smoke tests became unstable, because we are using && operator to install redoc dep: yarn install && yarn add redoc. Using eval i can support this operator

…jobs

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9fa6b3c. Configure here.}

chore: add resolutions to smoke tests

1ab2f6b

AlbinaBlazhko17 added 10 commits April 22, 2026 15:18

chore: add overrides

a4fe0f6

chore: add pnpm overrides

7396cdd

feat: add scipt to pull current installed version of config

cb57a7c

chore: move script into separate file

a120ae3

chore: remove pnpm overrides

3e06e62

feat: add different fix to pack config from node_modules

e7827c6

feat: migrate from yarn add to yarn install

da20af1

chore: fix config version script

1dcd7fb

fix: failing smoke tests

9fff599

chore: remove packing config

2771801

AlbinaBlazhko17 changed the title ~~chore: add resolutions to smoke tests~~ chore: fix flaky smoke tests on yarn Apr 22, 2026

AlbinaBlazhko17 marked this pull request as ready for review April 22, 2026 14:33

AlbinaBlazhko17 requested a review from a team as a code owner April 22, 2026 14:33

tatomyr reviewed Apr 23, 2026

View reviewed changes

AlbinaBlazhko17 and others added 2 commits April 23, 2026 10:13

chore: remove --no-lockfile arg

3c5609b

Merge branch 'main' into chore/fix-flaky-smoke-tests

daa1c70

cursor Bot reviewed Apr 23, 2026

View reviewed changes

Comment thread .github/workflows/smoke.yaml Outdated

AlbinaBlazhko17 added 4 commits April 23, 2026 10:16

chore: remove letter

64da1cd

feat: add to npm tests the same retry pattern as for yarn

0e709bc

chore: replace with the main

66f5c91

feat: add pining versions for config and ajv and use only for failed …

8ff38ce

…jobs

cursor Bot reviewed Apr 23, 2026

View reviewed changes

Comment thread tests/smoke/basic/pin-intersecting-deps.sh

AlbinaBlazhko17 added 2 commits April 23, 2026 16:01

chore: remove letter

54aa739

feat: add per job invocation

df6eff9

cursor Bot reviewed Apr 23, 2026

View reviewed changes

Comment thread .github/workflows/smoke.yaml Outdated

feat: add missing pin

af87090

cursor Bot reviewed Apr 23, 2026

View reviewed changes

Comment thread .github/workflows/smoke.yaml

AlbinaBlazhko17 and others added 3 commits April 23, 2026 16:35

feat: add pining to failed smoke

b523925

feat: add to failed smoke test pinning

41e177e

Merge branch 'main' into chore/fix-flaky-smoke-tests

f7c6af9

cursor Bot reviewed Apr 23, 2026

View reviewed changes

Comment thread .github/workflows/smoke.yaml

AlbinaBlazhko17 added 2 commits April 23, 2026 17:01

feat: add pining to the failed smoke test

388d798

feat: remove workaround to test if it works

bde52ed

cursor Bot reviewed Apr 23, 2026

View reviewed changes

Comment thread .github/workflows/smoke.yaml Outdated

AlbinaBlazhko17 added 3 commits April 23, 2026 18:21

feat: add to missing smoke test retry pattern

801cb4a

chore: remove unused overrides

82104c8

feat: add script to add redoc in packages and use for specific steps

9fa6b3c

cursor Bot reviewed Apr 24, 2026

View reviewed changes

Comment thread tests/smoke/basic/run-smoke.sh Outdated

AlbinaBlazhko17 and others added 2 commits April 24, 2026 11:50

chore: remove eval

d39f863

Merge branch 'main' into chore/fix-flaky-smoke-tests

92985f2

tatomyr approved these changes Apr 24, 2026

View reviewed changes

AlbinaBlazhko17 merged commit 0778679 into main Apr 24, 2026
45 checks passed

AlbinaBlazhko17 deleted the chore/fix-flaky-smoke-tests branch April 24, 2026 10:20

		@@ -0,0 +1,16 @@
		#!/bin/bash

		REDOCLY_CONFIG_VERSION=$(node -e "const fs = require('fs'); const p = JSON.parse(fs.readFileSync('./packages/core/package.json', 'utf8')); console.log(p.dependencies['@redocly/config'].replace(/\^/, ''));")

Conversation

AlbinaBlazhko17 commented Apr 22, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What/Why/How?

Reference

Testing

Screenshots (optional)

Check yourself

Security

Uh oh!

changeset-bot Bot commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

github-actions Bot commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

github-actions Bot commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AlbinaBlazhko17 commented Apr 22, 2026 •

edited by cursor Bot

Loading

changeset-bot Bot commented Apr 22, 2026 •

edited

Loading

github-actions Bot commented Apr 22, 2026 •

edited

Loading

github-actions Bot commented Apr 22, 2026 •

edited

Loading