Skip to content

ci: run the CLI suite - #923

Open
ozymandiashh wants to merge 1 commit into
getagentseal:feat/core-extractionfrom
ozymandiashh:fix/ci-run-cli-suite
Open

ci: run the CLI suite#923
ozymandiashh wants to merge 1 commit into
getagentseal:feat/core-extractionfrom
ozymandiashh:fix/ci-run-cli-suite

Conversation

@ozymandiashh

@ozymandiashh ozymandiashh commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Merge after #921. That PR fixes the aged fixture and adds the retry that keeps this job stable; landing this first would put a red, flaky gate in front of every subsequent PR.

CI followed the core package through the workspace move: there is a job for typecheck, test, build, verify-dist and pack across three node versions. Nothing runs the CLI's suite — roughly 2470 tests, including the 29 provider bridge suites that byte-compare against goldens captured before the extraction.

That matters more than a missing job usually would. #809 states that every phase PR passes a byte-identical parity gate. The gate exists and is real — packages/cli/tests/providers/*-bridge.test.ts are genuine byte-compare tests — but nothing has been running it, so "parity holds" has been an assertion rather than a check for the whole extraction.

Two things stood in the way

"test": "vitest" is watch mode. In CI that hangs the runner instead of failing. Now vitest run.

The root test script only forwarded to the CLI workspace, so core's guardrail suite — architecture-gate, content-smuggling, import-smoke, schema-drift — never ran from the command a contributor actually types. Worse, the CLI half could not run from a clean checkout at all: core's exports resolve to dist, which is gitignored, so a fresh clone died on Cannot find module '@codeburn/core'.

The root script now builds core first. That costs a build on every local run, which is a genuine annoyance; the alternative is a script that only works if you happened to build core earlier.

Two node legs, and why that is not sprawl

A review caught something that would have made this job worse than useless: zed-bridge.test.ts seeds its fixture with zlib's zstd, which only landed in Node 22.15, so it self-skips below that. Pinning the job to the engines floor of 22.13 would have silently skipped a byte-compare parity suite and reported green — precisely the failure mode this PR exists to eliminate.

So the job runs two legs. The floor leg keeps the >=22.13 promise honest; the 24.x leg makes the parity gate actually execute. Worth flagging separately: the engines floor and that test's real requirement disagree, which is a question for you rather than something to paper over here.

Other review fixes

  • 15-minute timeout. Actions defaults to 360; measured wall time for the suite is about 2 minutes. A hung run cannot burn a runner for six hours.
  • Workspace-versions check moved before npm ci. It only reads the manifests and the lockfile, so drift now fails in a second rather than after a full install.

Verification

actionlint clean on the workflow. Verified that building core is sufficient: no CLI test spawns the built CLI, and none depend on the dash build or the litellm bundle. The Playwright sync E2E self-skips without three env vars, so it stays inert in CI — noted rather than fixed.

The repo moved to npm workspaces and CI followed the core package: there is a
job for typecheck, test, build, verify-dist and pack across three node
versions. Nothing runs the CLI's suite — roughly 2470 tests, including the 29
provider bridge suites that byte-compare against goldens captured before the
extraction. Issue getagentseal#809 says every phase PR passes a byte-identical parity gate;
until now nothing enforced it.

Two things stood in the way. The CLI's test script was `vitest`, which is watch
mode — in CI that hangs a runner instead of failing. And the root test script
forwarded only to the CLI workspace, so core's guardrail suite never ran from
the command a contributor reaches for, while the CLI half of it could not run
from a clean checkout at all: core's exports resolve to dist, which is
gitignored.

The root script now builds core before running either suite. That costs a
build on every local run, which is a real annoyance, but the alternative is a
script that only works if you happen to have built core earlier.

The CLI job runs the suite on two node lines: the engines floor (22.13.x) and
24.x. The second leg is not matrix sprawl — the zed bridge parity suite seeds
its fixture with zlib's zstd, which only landed in 22.15, so a floor-only job
would silently skip the byte-compare gate this job exists to run and still go
green. The floor leg keeps the >=22.13 promise enforced; the 24.x leg makes
the parity gate actually execute. The job carries a 15-minute timeout —
measured wall time for the whole suite is ~2 minutes — so a hung run cannot
burn a runner for the default six hours, and the workspace-versions check runs
before npm ci: it only reads the three manifests and the lockfile, so a drift
fails in a second instead of after a full install.
@ozymandiashh
ozymandiashh force-pushed the fix/ci-run-cli-suite branch from affdc6f to 4a971ee Compare August 5, 2026 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant