ci: gate the browser-bridge node suite in the flake (460 tests, counted) - #280
Merged
Conversation
The 460-test `.mjs` suite gated NOTHING. flake.nix had zero references to
node/mjs; `checks.pytests` runs scripts/run-tests.sh, which invokes
`python -m pytest` only. Every "460 pass" claim in recent PRs was a MANUAL
run, and a .mjs regression could merge freely.
Adds `checks.x86_64-linux.nodetests` + scripts/run-node-tests.sh.
Separate check, not folded into pytests: distinct toolchain (so the python
gate's closure doesn't grow node, and a node bump can't invalidate its
cache), distinct failure signal from `nix flake check`, and they run in
parallel so the wall-clock cost is ~0.
Two structural guards in the runner, because a green exit code lies here:
1. Never pass a directory. `node --test <dir>` silently collapses to
`# tests 1 / # fail 1` (MODULE_NOT_FOUND). The runner globs the files
into an array and refuses any element that is not a regular file
ending in .test.mjs, so it cannot degrade to the dir form.
2. COUNT the tests. Parses node's TAP summary and fails below MIN_TESTS
(450; 460 today) and MIN_FILES (14). Without this the gate can go
green while testing nothing -- exactly how the pytest gate silently
skipped 41 test_server.py tests for want of curl on PATH.
Hermeticity audited AND measured: the suite has no createServer/.listen(),
no spawn/execFile, every fetch is a stub on globalThis.fetch, and its only
I/O is repo-relative reads plus tmpdir() writes. All 460 run in the
sandbox -- nothing excluded, nothing skipped.
nodejs is pinned from this flake's nixpkgs: the sandbox runs v24.18.0 while
the host PATH node is v26.5.0, and the runner echoes node --version each
run so the toolchain is never a guess.
run-tests.sh's header claimed to be the source of truth for the flake gate;
it now states it covers the PYTHON suites only and points at the node
runner + check.
Verification:
* negative control: mutating one assertion in browser_tool.test.mjs made
`nix build .#checks.x86_64-linux.nodetests` exit 1 with
"files=14 tests=460 pass=459 fail=1 / RESULT: FAIL". Reverted -> exit 0.
* runner guards proven reachable individually: MIN_TESTS=99999 -> exit 1
with the vacuous-green error; MIN_FILES=999 -> exit 2; no node on PATH
-> exit 2.
* `nix flake check` -> "running 2 flake checks ... all checks passed!"
* pytests unchanged and still green.
Timing: nodetests ~8s. `nix flake check` 3m30s vs 3m28s for pytests alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev
added a commit
that referenced
this pull request
Aug 2, 2026
…ollapses (#284) The flake's pytest gate read an exit code and counted nothing about skips, while carrying 125 skipped tests. A skipped test reports safety without providing it — and that exact failure already shipped once (41 test_server.py tests skipping for want of `curl`, fixed in #251 with nothing asserting the fix kept working). Mirrors what #280 did for the node suite. scripts/run-tests.sh now parses pytest's summary instead of its exit code and enforces four guards: 1. REQUIRED_TOOLS — the 10 binaries the suites `skipif` on (curl, node, rg, git, awk, jq, grep, setsid, bash, python3) must be on PATH; absence is a named FATAL, not 41 silent skips. 2. EXPECTED_SKIPS — a PINNED (directory, reason-regex) set, not a numeric ceiling. Every skip must match an entry AND the skip total must equal the entry count, so a new skip, a relocated skip, a swapped skip and a pinned skip that stops firing all fail. Suites run with -rs so reasons print. 3. Collected-test floors — global MIN_TESTS=2850 (2920 today) plus a >=1 per-directory floor, so one suite collapsing cannot hide in the total. 4. Parse guard — an unparseable summary is a FAILURE, not a pass. flake.nix: adds pkgs.nodejs to checks.pytests. MEASURED — the initiatives suite reported `660 passed, 123 skipped` in the sandbox vs `783 passed, 0 skipped` on a host with node; all 123 are recovered. Sandbox skips go 125 -> 2, both pinned and both genuinely unavoidable there. Verified — each guard broken individually and shown to fail with ITS OWN message: tool precondition (drop pkgs.curl -> "FATAL — required tool(s) missing from PATH: curl"); skip pin (drop curl AND bypass guard 1 -> "76 UNPINNED skip group(s)" + "111 test(s) skipped, but 2 are pinned"); global floor (trim to one dir -> "only 331 tests were collected, floor is 2850", alone); per-dir floor (a dir with no tests, MIN_TESTS=1 -> "nix (collected 0 tests)", alone); parse guard (a conftest ImportError -> "unparseable summary", alone). Green after each revert. `nix flake check` green end to end in 3:36 — pytests 2920 collected / 2918 passed / 2 skipped, nodetests 468/468. Added test time is ~5s (initiatives 15.1s -> 20.5s from the recovered node tests). Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 2, 2026
ZacxDev
added a commit
that referenced
this pull request
Aug 2, 2026
… clawgate) RULES.md — three additions, all from measured ground cases: * Verify the CONSUMER is running your artifact, not that the deploy reported success. `ship.sh` reported "VERIFIED — on branch main at origin/main + switched" while the browser-bridge unit was crash-looping on `OSError: [Errno 98] Address already in use`; an orphaned process from the previous day (Aug 1 16:18, in NO systemd cgroup) held 127.0.0.1:8788 and served the OLD server.py. The converge check verified branch + switch and structurally could not see that the service never started. Recipe (`ss -lptn` -> `/proc/<pid>/cgroup` -> `systemctl show -p MainPID`) was executed live before being written. * A count of DECLARATIONS is not a count of INSTANCES. A grep of `skipif` decorators found "2 node-related skips"; the two decorators gated 123 tests (initiatives: 660 passed/123 skipped sandboxed vs 783 passed/0 skipped with node). A 60x sizing error. * Sharpen the harness bullet from a MANIFESTATION to a CLASS: when you parse a tool's OUTPUT, its format is an unpinned dependency, and "no matches" means "possibly the wrong pattern". The file already named `diff`'s unified default; that rule was READ this session and the trap was hit anyway in three new shapes (a false CLEAN over a 1,445-byte difference that only `cmp` caught; node 24's reporter change emptying a `^# (tests|pass|fail)` grep; `rc=$?` reading `echo`'s status). CLAUDE.md — correct stale facts in the browser-bridge bullet: * Drop the hand-pinned "281 B free today". test_skill_size.py owns MAX_BYTES/MIN_HEADROOM_BYTES (the floor was raised to 250 in #275); point at it instead of re-pinning a figure that rots. * Record that CI now gates BOTH suites — nodetests (#280) and the pytest gate's silent-coverage-collapse guards (#284) — and the headline outcome, skips 125 -> 2. * Extension 0.7.1; `nav`/`open` accept `--wake[=MS]`. reference/tabs-instances.md — record the #273 measurement: a forced discard assigns a NEW tabId and releases ownership (484065264 -> 484065273, ownedTabId None), so the stale-documentEmulation hazard is not reachable on this Chromium. Scope stated honestly (one build, one profile, one mechanism; auto-discard not exercised; onReplaced inferred, not observed) and the load-bearing caveat kept: the safety is a property of the BROWSER, not of the bridge. Placed in reference/, not the byte-capped core — SKILL.md is unchanged at 11,845 B. clawgate/SKILL.md — live version 0.7.82 (embedded kubeclaw chart 0.7.1, so the pending re-sync note is resolved); "derive from the LIVE pin" stays primary. Adds the `make check-chart` hazard: it depends on `sync-chart`, which rsyncs from ~/workspace/kubeclaw — that clone sat at 0.3.14 against a vendored 0.7.1, so running it would have clobbered the deployed chart and reported a false failure. Fetch + `merge --ff-only` that clone first. Docs-only. No source file touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev
added a commit
that referenced
this pull request
Aug 2, 2026
…ations-vs-instances + output-format class, CLAUDE.md CI/byte facts, #273 tab discard, clawgate check-chart) (#287) * docs: record what 2026-08-02 measured (RULES, CLAUDE.md, tab discard, clawgate) RULES.md — three additions, all from measured ground cases: * Verify the CONSUMER is running your artifact, not that the deploy reported success. `ship.sh` reported "VERIFIED — on branch main at origin/main + switched" while the browser-bridge unit was crash-looping on `OSError: [Errno 98] Address already in use`; an orphaned process from the previous day (Aug 1 16:18, in NO systemd cgroup) held 127.0.0.1:8788 and served the OLD server.py. The converge check verified branch + switch and structurally could not see that the service never started. Recipe (`ss -lptn` -> `/proc/<pid>/cgroup` -> `systemctl show -p MainPID`) was executed live before being written. * A count of DECLARATIONS is not a count of INSTANCES. A grep of `skipif` decorators found "2 node-related skips"; the two decorators gated 123 tests (initiatives: 660 passed/123 skipped sandboxed vs 783 passed/0 skipped with node). A 60x sizing error. * Sharpen the harness bullet from a MANIFESTATION to a CLASS: when you parse a tool's OUTPUT, its format is an unpinned dependency, and "no matches" means "possibly the wrong pattern". The file already named `diff`'s unified default; that rule was READ this session and the trap was hit anyway in three new shapes (a false CLEAN over a 1,445-byte difference that only `cmp` caught; node 24's reporter change emptying a `^# (tests|pass|fail)` grep; `rc=$?` reading `echo`'s status). CLAUDE.md — correct stale facts in the browser-bridge bullet: * Drop the hand-pinned "281 B free today". test_skill_size.py owns MAX_BYTES/MIN_HEADROOM_BYTES (the floor was raised to 250 in #275); point at it instead of re-pinning a figure that rots. * Record that CI now gates BOTH suites — nodetests (#280) and the pytest gate's silent-coverage-collapse guards (#284) — and the headline outcome, skips 125 -> 2. * Extension 0.7.1; `nav`/`open` accept `--wake[=MS]`. reference/tabs-instances.md — record the #273 measurement: a forced discard assigns a NEW tabId and releases ownership (484065264 -> 484065273, ownedTabId None), so the stale-documentEmulation hazard is not reachable on this Chromium. Scope stated honestly (one build, one profile, one mechanism; auto-discard not exercised; onReplaced inferred, not observed) and the load-bearing caveat kept: the safety is a property of the BROWSER, not of the bridge. Placed in reference/, not the byte-capped core — SKILL.md is unchanged at 11,845 B. clawgate/SKILL.md — live version 0.7.82 (embedded kubeclaw chart 0.7.1, so the pending re-sync note is resolved); "derive from the LIVE pin" stays primary. Adds the `make check-chart` hazard: it depends on `sync-chart`, which rsyncs from ~/workspace/kubeclaw — that clone sat at 0.3.14 against a vendored 0.7.1, so running it would have clobbered the deployed chart and reported a false failure. Fetch + `merge --ff-only` that clone first. Docs-only. No source file touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(rules): two-tier suites must be green in BOTH; three declarations-vs-instances instances Both measured while unbreaking the pytest gate (#289). TWO TIERS. 'Gate on the merged tree' extended to environments. The same suite runs in the nix sandbox and on a dev host, and each environment silently decides which tests execute — so a defect can be permanently unobservable in the tier you happen to read. #276 shipped THREE regressions that masked each other: (1) a FILE in run-tests.sh's target list rejected by `[ ! -d ]` -> gate red, 913 tests never ran (2) SECRET_PATTERNS moved to guard_core.py -> the drift test parses [] and FAILS on a host with the hook deployed, SKIPS in the sandbox (3) 10 nix-instantiate tests pytest.fail() without the binary -> FAIL in the sandbox, pass on every dev host (2) and (3) are exact complements; both hid behind (1)'s red. DECLARATIONS VS INSTANCES gets its third instance in one session: 2 skipif decorators -> 123 tests; 1 list entry -> 913 tests; 1 nix_eval() helper -> 10 parametrized tests. Three is a pattern. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ci: gate the browser-bridge node suite in the flake (460 tests, counted)
The 460-test
.mjssuite gated NOTHING. flake.nix had zero references tonode/mjs;
checks.pytestsruns scripts/run-tests.sh, which invokespython -m pytestonly. Every "460 pass" claim in recent PRs was a MANUALrun, and a .mjs regression could merge freely.
Adds
checks.x86_64-linux.nodetests+ scripts/run-node-tests.sh.Separate check, not folded into pytests: distinct toolchain (so the python
gate's closure doesn't grow node, and a node bump can't invalidate its
cache), distinct failure signal from
nix flake check, and they run inparallel so the wall-clock cost is ~0.
Two structural guards in the runner, because a green exit code lies here:
node --test <dir>silently collapses to# tests 1 / # fail 1(MODULE_NOT_FOUND). The runner globs the filesinto an array and refuses any element that is not a regular file
ending in .test.mjs, so it cannot degrade to the dir form.
(450; 460 today) and MIN_FILES (14). Without this the gate can go
green while testing nothing -- exactly how the pytest gate silently
skipped 41 test_server.py tests for want of curl on PATH.
Hermeticity audited AND measured: the suite has no createServer/.listen(),
no spawn/execFile, every fetch is a stub on globalThis.fetch, and its only
I/O is repo-relative reads plus tmpdir() writes. All 460 run in the
sandbox -- nothing excluded, nothing skipped.
nodejs is pinned from this flake's nixpkgs: the sandbox runs v24.18.0 while
the host PATH node is v26.5.0, and the runner echoes node --version each
run so the toolchain is never a guess.
run-tests.sh's header claimed to be the source of truth for the flake gate;
it now states it covers the PYTHON suites only and points at the node
runner + check.
Verification:
nix build .#checks.x86_64-linux.nodetestsexit 1 with"files=14 tests=460 pass=459 fail=1 / RESULT: FAIL". Reverted -> exit 0.
with the vacuous-green error; MIN_FILES=999 -> exit 2; no node on PATH
-> exit 2.
nix flake check-> "running 2 flake checks ... all checks passed!"Timing: nodetests ~8s.
nix flake check3m30s vs 3m28s for pytests alone.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
🤖 Generated with Claude Code