From 9669d9c7a2ac44f4c6ff06b345ccd2741a552d91 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 6 Aug 2026 12:47:11 +0000 Subject: [PATCH 1/2] docs: correct CI facts and add the missing test tiers to the testing strategy The testing strategy is the doc that answers "how is behavior verified", so a wrong operational assumption there is load-bearing. Three claims no longer hold: - It described a self-hosted GitHub Actions runner that must provide tmux, but every workflow job is `runs-on: ubuntu-latest` and the tmux-dependent ones install tmux themselves via apt-get. The `CREW_REQUIRE_TMUX=1` hard-failure guarantee is unchanged and is kept verbatim in meaning. - Package verification claimed pack-smoke proves Role/Team resolution through `import.meta.url`, but templates are compiled-in string constants and resolve no runtime path (src/templates.ts); what pack-smoke actually exercises is `init` seeding them from the installed tarball. - The layer table stopped at package smoke, omitting the jsdom `web`/`docs` vitest projects (on every PR, outside the 95% gate) and the Playwright Console smoke run by ui-e2e.yml. CONTRIBUTING's two-tier CI list likewise never named ui-e2e.yml. Closes #33 Co-Authored-By: Claude Opus 5 --- CONTRIBUTING.md | 4 ++++ docs/design/testing-strategy.md | 35 ++++++++++++++++++++++----------- 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b7195ac..a803255 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -54,6 +54,10 @@ pushing. fake Participant CLI (the `launch-e2e` label). Locally it runs only when `CREW_LAUNCH_E2E=1` is set, so a normal per-PR `npm test` shows it as a visible skip instead of silently leaving it out. + - `ui-e2e.yml` — the Console dashboard browser smoke, which builds the real dashboard + bundle, starts `crew ui`, and drives it in Chromium with Playwright (the `ui-e2e` + label). It is not part of `npm test`; locally you run it on demand with + `npm run e2e:ui`. If a full-tier run fails only some of the time, treat that as a release blocker. Do not just rerun it and hope it passes. diff --git a/docs/design/testing-strategy.md b/docs/design/testing-strategy.md index 495cab8..cae1c50 100644 --- a/docs/design/testing-strategy.md +++ b/docs/design/testing-strategy.md @@ -11,9 +11,11 @@ state-machine behavior. | Unit | validators, formatters, Role/Team/platform registry, launch plan | in-process | fast, exhaustive edge cases | | Store integration | Store domain operations against temp SQLite | in-process, real DB | schema constraints, transitions, migrations, transactions | | Program integration | `run(argv, io)` | in-process | parsing, exit/error/output contracts for every command | +| Browser component | Console dashboard and documentation-site modules in jsdom | in-process, jsdom | rendering, view-model, and routing behavior without a real browser | | Spawn | built `crew` executable | many OS processes | locking, stdout/stderr, shebang, signals, crash behavior | | Launcher contract | Launcher with recording process adapter | in-process | the exact subprocess argv, call order, cleanup, no shell interpretation | | tmux e2e | built executable + real tmux | real processes | pane readiness, Relay nudge, session cleanup | +| Console browser e2e | built dashboard bundle served by `crew ui` | real server + Chromium | the board renders seeded Workspace data and Operator actions apply | | Platform smoke | installed Participant CLI | release-only/manual or isolated job | generated artifact discovery and a finite workflow | | Package smoke | packed tarball in clean temp prefix | real install | published file list, templates, executable, runtime floor | @@ -21,13 +23,23 @@ The spawn layer starts many real operating-system processes on purpose, so it ca behavior — locking, crashes, signals — that in-process tests cannot reach. "e2e" means end-to-end: the whole path is exercised, from the built executable to a real tmux session. +The browser-component layer is two vitest projects, `web` (the Console dashboard under +`web/`) and `docs` (the documentation site under `docs-site/`), both running in jsdom. They +run on every PR as part of `npm run test:coverage`, but they stay outside the 95% coverage +gate, which measures `src/**` and `bin/**` only. The Console browser e2e is deliberately off +the PR critical path: `npm run e2e:ui` drives the Playwright specs under `e2e/ui/` against a +real Chromium, and `.github/workflows/ui-e2e.yml` runs them nightly, on manual dispatch, and +on a PR only when it carries the `ui-e2e` label. A flaky failure there is a release failure, +not a candidate for a blind retry. + ## Test environment matrix -- Self-hosted GitHub Actions runner, Node `24.18.0`. -- CI requires the self-hosted runner to provide tmux and runs the test suite with - `CREW_REQUIRE_TMUX=1`, so the real-tmux e2e always runs in CI and a missing tmux is a hard - failure (never a silent skip). Locally the real-tmux e2e gracefully skips when tmux is - absent; the recording-adapter tests remain mandatory everywhere. +- GitHub-hosted GitHub Actions runners (`ubuntu-latest`), Node `24.18.0`. +- The tmux-dependent workflows install tmux themselves (`apt-get install -y tmux`, then + `tmux -V` to prove it) and run the test suite with `CREW_REQUIRE_TMUX=1`, so the real-tmux + e2e always runs in CI and a missing tmux is a hard failure (never a silent skip). Locally + the real-tmux e2e gracefully skips when tmux is absent; the recording-adapter tests remain + mandatory everywhere. - Windows may run the core Program/Store tests as informational until officially supported. Tests use an isolated temporary `HOME`, Workspace, XDG variables, Git repo, and database. No @@ -114,11 +126,11 @@ This suite forces many processes to hit the database at the same time. All child wait on a shared start barrier so their operations genuinely overlap. One knob, `CREW_STRESS_ITERS`, scales every statistical case together (the deterministic CONTENTION/lock and crash cases run once); the two CI tiers run a fast 25 iterations per case -on every PR and the full 500 per case nightly and at release on the self-hosted runner at -Node `24.18.0`. The local `npm run test:stress` convenience script defaults to an -intermediate 300 per case (override via `CREW_STRESS_ITERS`) as a quick pre-push check -between those tiers. The random retry delay is seeded per child from `CREW_STRESS_SEED` -(default 1), so a run can be replayed, apart from OS scheduling differences. +on every PR and the full 500 per case nightly and at release at Node `24.18.0`. The local +`npm run test:stress` convenience script defaults to an intermediate 300 per case (override +via `CREW_STRESS_ITERS`) as a quick pre-push check between those tiers. The random retry +delay is seeded per child from `CREW_STRESS_SEED` (default 1), so a run can be replayed, +apart from OS scheduling differences. 1. **Same-id join:** N processes request `worker`; the result is exactly N unique ids with no gaps in the claimed suffix range and no claim that reported success but did not stick. @@ -231,7 +243,8 @@ temporary prefix and verify: - `crew --version` and `crew --help`; - init + join + send + receive + reviewed Task flow; -- packaged Role/Team resolution through `import.meta.url`; +- the Role/Team templates, which ship as compiled-in string constants (`src/templates.ts`) + with no runtime path resolution, seed a Workspace through the packed executable's `init`; - the executable bit and shebang line on macOS and Linux; - Node below the engine floor fails with a clear message; - no source maps or test fixtures leak secrets or local paths. From c4f27d3a431804fe567a09f387497fe3b8921205 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 6 Aug 2026 15:21:56 +0000 Subject: [PATCH 2/2] docs: state only verified pack-smoke facts and correct the stress trigger MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review follow-up on the same #33 drift pass: - The rewritten package-verification bullet sits under "verify:", but pack-smoke never asserts template seeding — its `init` case only checks the {type:'init',schema_version:1} envelope, and the following `join --role worker` proves nothing because resolveRole falls back to PACKAGED_ROLES. Restate the bullet as what is actually checked: the compiled-in-constant design and the packed-file allowlist that proves no template ships as a separate asset. - The heavy stress tier does not run "at release": stress.yml has only schedule, workflow_dispatch, and a labeled-PR trigger, and publish.yml runs the fast tier (CREW_STRESS_ITERS=25). Say nightly and manual dispatch during release prep. - The browser-e2e sentence declared a release gate; product-spec.md owns the canonical release-gate table and forbids separate copies, so reference the existing full-tier convention instead. - Use CLAUDE.md's "GitHub-hosted runners" phrasing, and drop the partial apt-get command that would fail on a cold package index. - CONTRIBUTING: `npm run e2e:ui` needs a browser, which `npm ci` does not install; name the `playwright install` step the workflow itself runs. Co-Authored-By: Claude Opus 5 --- CONTRIBUTING.md | 1 + docs/design/testing-strategy.md | 32 +++++++++++++++++--------------- 2 files changed, 18 insertions(+), 15 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a803255..008faf0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -57,6 +57,7 @@ pushing. - `ui-e2e.yml` — the Console dashboard browser smoke, which builds the real dashboard bundle, starts `crew ui`, and drives it in Chromium with Playwright (the `ui-e2e` label). It is not part of `npm test`; locally you run it on demand with + `npx playwright install --with-deps chromium` once to get the browser, then `npm run e2e:ui`. If a full-tier run fails only some of the time, treat that as a release blocker. Do not diff --git a/docs/design/testing-strategy.md b/docs/design/testing-strategy.md index cae1c50..77226ef 100644 --- a/docs/design/testing-strategy.md +++ b/docs/design/testing-strategy.md @@ -29,17 +29,17 @@ run on every PR as part of `npm run test:coverage`, but they stay outside the 95 gate, which measures `src/**` and `bin/**` only. The Console browser e2e is deliberately off the PR critical path: `npm run e2e:ui` drives the Playwright specs under `e2e/ui/` against a real Chromium, and `.github/workflows/ui-e2e.yml` runs them nightly, on manual dispatch, and -on a PR only when it carries the `ui-e2e` label. A flaky failure there is a release failure, -not a candidate for a blind retry. +on a PR only when it carries the `ui-e2e` label. Like the other full-tier suites, an +intermittent failure there is investigated rather than blindly retried. ## Test environment matrix -- GitHub-hosted GitHub Actions runners (`ubuntu-latest`), Node `24.18.0`. -- The tmux-dependent workflows install tmux themselves (`apt-get install -y tmux`, then - `tmux -V` to prove it) and run the test suite with `CREW_REQUIRE_TMUX=1`, so the real-tmux - e2e always runs in CI and a missing tmux is a hard failure (never a silent skip). Locally - the real-tmux e2e gracefully skips when tmux is absent; the recording-adapter tests remain - mandatory everywhere. +- GitHub-hosted runners (`ubuntu-latest`), Node `24.18.0`. +- The tmux-dependent workflows install tmux themselves via `apt-get` (then `tmux -V` to prove + it) and run the test suite with `CREW_REQUIRE_TMUX=1`, so the real-tmux e2e always runs in + CI and a missing tmux is a hard failure (never a silent skip). Locally the real-tmux e2e + gracefully skips when tmux is absent; the recording-adapter tests remain mandatory + everywhere. - Windows may run the core Program/Store tests as informational until officially supported. Tests use an isolated temporary `HOME`, Workspace, XDG variables, Git repo, and database. No @@ -126,11 +126,11 @@ This suite forces many processes to hit the database at the same time. All child wait on a shared start barrier so their operations genuinely overlap. One knob, `CREW_STRESS_ITERS`, scales every statistical case together (the deterministic CONTENTION/lock and crash cases run once); the two CI tiers run a fast 25 iterations per case -on every PR and the full 500 per case nightly and at release at Node `24.18.0`. The local -`npm run test:stress` convenience script defaults to an intermediate 300 per case (override -via `CREW_STRESS_ITERS`) as a quick pre-push check between those tiers. The random retry -delay is seeded per child from `CREW_STRESS_SEED` (default 1), so a run can be replayed, -apart from OS scheduling differences. +on every PR and the full 500 per case nightly and on manual dispatch during release prep, at +Node `24.18.0`. The local `npm run test:stress` convenience script defaults to an +intermediate 300 per case (override via `CREW_STRESS_ITERS`) as a quick pre-push check +between those tiers. The random retry delay is seeded per child from `CREW_STRESS_SEED` +(default 1), so a run can be replayed, apart from OS scheduling differences. 1. **Same-id join:** N processes request `worker`; the result is exactly N unique ids with no gaps in the claimed suffix range and no claim that reported success but did not stick. @@ -243,8 +243,10 @@ temporary prefix and verify: - `crew --version` and `crew --help`; - init + join + send + receive + reviewed Task flow; -- the Role/Team templates, which ship as compiled-in string constants (`src/templates.ts`) - with no runtime path resolution, seed a Workspace through the packed executable's `init`; +- no Role/Team template ships as a separate asset: templates are compiled-in string + constants (`src/templates.ts`) with no runtime path resolution, and the packed-file + allowlist admits only `dist/**/*.js`, the bundled Console page, `README.md`, `LICENSE`, + and `package.json`; - the executable bit and shebang line on macOS and Linux; - Node below the engine floor fails with a clear message; - no source maps or test fixtures leak secrets or local paths.