Testing and QA

Testing and QA

The test pyramid (v1.156.0 baseline)

Layer	Count	Command	What it covers
Unit / integration	2396	npm test (node --test tests/*.test.mjs tests/acceptance/*.test.mjs)	In-process createApp() hit with fetch on an ephemeral port; registry invariants; sanitizers; provider parity suites (stubbed transport, no network)
Playwright browser	70+	npm run test:e2e:browser	Smoke, full-cycle, forms, locale sweep, theme toggle, persistent widgets (docs FAB + usage HUD)
Smoke E2E	20	npm run test:e2e (tests/e2e.mjs)	Real server in a child process, every route walked
Comprehensive E2E	23	npm run test:e2e:full (tests/e2e-comprehensive.mjs)	Long full-surface pass — catches SPA regressions unit tests can't

• Coverage floor: 80 % on non-trivial logic; the actual baseline is ~93 % line / ~83 % branch (npm run test:coverage).
• Never hardcode port 4317 in tests — always server.listen(0).
• TDD when adding behavior (red → green → refactor); skipped only for pure refactors already under full coverage.
• No mocks of internal collaborators — fake the parent by pointing CAREER_OPS_ROOT at a mktemp -d with the minimal files the path under test needs.

CI gates

npm run test:ci = npm test + three deterministic gates:

1. scripts/check-no-also-leftovers.mjs — doc-hygiene check.
2. scripts/check-changelog-parity.mjs — all 16 translated CHANGELOGs at the current version.
3. tools/i18n-audit.mjs — locale dictionary integrity.

Plus in the workflow matrix: Node 18/20/22 runs, CodeQL, and the Playwright suites. Key i18n/docs parity tests: tests/i18n-locale-files.test.mjs, tests/i18n-coverage.test.mjs (snapshot at tests/fixtures/i18n-dict.snapshot.json), tests/canonical-docs-coverage.test.mjs, tests/help-ru-config-section.test.mjs, tests/help-ui.test.mjs, tests/manifesto-link.test.mjs (help bundle 29 H2 / 105 H3 structure gates).

The hard gate is ci.yml, not the pre-commit hook. The pre-commit AI review is advisory; a green pre-commit with a red CI is possible — watch the CI run.

CI-isolation rules (the traps)

Tests cannot assume the parent career-ops project exists. The two rules that cost releases when violated:

1. PATHS resolves once per process. server/lib/paths.mjs computes PROJECT_ROOT at import time. A test that sets CAREER_OPS_ROOT in before() must load every paths.mjs carrier (server/index.mjs, prompts.mjs, store.mjs, en-scanner.mjs, ru-scanner.mjs, paths.mjs) via dynamic import() inside before(). A top-level static import runs before the env is set, pins the REAL parent, and leaks writes (e.g. PUT /api/profile) into the user's real files. Guards: tests/paths-once.test.mjs, tests/test-root-isolation.test.mjs.
2. Bootstrap the minimal layout. CAREER_OPS_ROOT=$(mktemp -d) + write only what the test needs (cv.md, portals.yml, …). Fixtures live under tests/fixtures/ — never real user data.
3. New FS-write helpers take an explicit path param — otherwise the once-per-process resolution leaks writes into the real parent.

Two dedicated reviewer subagents enforce conventions: web-ui-route-reviewer (routes/security envelope) and test-isolation-reviewer (CI isolation, no live network, no port collisions).

QA prompt methodology (qa/)

Every release ships a QA regression prompt — qa/QA-REGRESSION-PROMPT-v<version>.md — a delta driver covering only what that release added, run on top of the definitive whole-project prompt qa/QA-REGRESSION-PROMPT.md. Structure of a delta driver (e.g. v1.118.0):

• A header pinning the version under test, parentVersion, route-module count, adapter count, and the unit-test baseline.
• One § per shipped feature with concrete, executable checks: exact endpoints to hit, exact test files to run, exact UI states to verify (e.g. "GET /api/scan/sources → the EN list contains all 9 new values", "POST /api/tracker {status:'Hired'} → row lands; unknown status degrades to Evaluated").
• A docs & i18n fan-out § (parity scripts + locale-file tests + help H2/H3 counts).
• A sign-off §: npm test green (≥ baseline), npm run test:ci green, Playwright green, both E2E suites green, CI matrix (Node 18/20/22 + CodeQL) green — with the known CodeQL false-positive dismissal rationale pre-stated.

Operational rules learned the hard way (see also Troubleshooting & FAQ):

• Never npm test 2>&1 | grep … — grep masks the exit code; two releases shipped failing tests this way. Run the suite, capture $?, grep separately.
• GET-only when smoke-testing a live deployed server — no write-API calls against a real parent.
• Baselines only ratchet up: the next ship must keep all four suite counts ≥ the previous floor.