Fix/watchdog deadman resume - #7
Merged
Merged
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d22ccc8409
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+1480
to
+1481
| " try { if (watchdogChild) { watchdogExpectedExit = true; watchdogChild.kill(); } } catch (e) {}", | ||
| " try { fs.unlinkSync(watchdogHaltPath); } catch (e) {}", |
There was a problem hiding this comment.
If the watchdog is killed after the last await but before this synchronous completion tail finishes, its exit callback remains queued; this code marks the death expected and deletes the persistent dead-man evidence before the callback can invoke haltOnDeadWatchdog. The manager then exits 0, permanently masking loss of the guard. Wait for and validate the watchdog's controlled shutdown before deleting these files rather than suppressing any pending exit.
Useful? React with 👍 / 👎.
| - name: List B — publication-hygiene scan (trusted context only; opt-in; never fork-PR) | ||
| # Runs only when the maintainer has configured the hygiene secrets. Without them | ||
| # the scan can't check anything, so skip (green) rather than hard-fail the build. | ||
| if: github.event_name != 'pull_request' && env.HYGIENE_HMAC_KEY != '' |
There was a problem hiding this comment.
When HYGIENE_HMAC_KEY is configured but HYGIENE_DIGESTS is missing or empty, this condition runs the supposedly opt-in step, and hygiene-scan.js --ci exits 2 because the digest secret is absent. Thus a partially or previously configured repository still gets a permanently red docs-hygiene job instead of the documented green skip; include the digest secret in the condition as well.
Useful? React with 👍 / 👎.
Comment on lines
+170
to
+173
| env: | ||
| # Exposed at job level so the List B step can skip itself when the maintainer | ||
| # has not configured the hygiene secrets (List B is opt-in; List A always gates). | ||
| HYGIENE_HMAC_KEY: ${{ secrets.HYGIENE_HMAC_KEY }} |
There was a problem hiding this comment.
When a pull request originates from a branch in the same repository, Actions supplies repository secrets, and this job-level environment makes HYGIENE_HMAC_KEY available to every step even though the List B step is skipped. A contributor can therefore modify the PR-controlled scripts/docs-lint.js executed at line 180 to read or exfiltrate the key; keep the secret scoped to a trusted non-PR step/job instead of placing it on a job that runs for pull_request.
Useful? React with 👍 / 👎.
Comment on lines
+55
to
+58
| "tests/gsa/", | ||
| "tests/gsa-mcp/", | ||
| "tests/gsa-register/", | ||
| "tests/register/" |
There was a problem hiding this comment.
These suites are now non-gating individually, but their replacement gate in tests/gauntlet/v0.3.0-battery.js only compares each suite's total FAIL count with a numeric allowance (lines 16-22 and 43-48). If a product change makes one documented false-positive pass while causing a different assertion to fail, the count remains within the allowance and CI reports HOLD despite the new regression. This affects several newly exempted security/attestation suites with nonzero allowances; whitelist the exact adjudicated test names rather than relying on aggregate counts before making their raw exits non-gating.
Useful? React with 👍 / 👎.
Comment on lines
+55
to
+58
| "tests/gsa/", | ||
| "tests/gsa-mcp/", | ||
| "tests/gsa-register/", | ||
| "tests/register/" |
There was a problem hiding this comment.
In GitLab mirrors using ci-templates/gitlab-ci.yml, these prefixes make all affected failures evidence-only, but the template's test script ends after the manifest-driven runner and never invokes tests/gauntlet/v0.3.0-battery.js. Unlike the inspected GitHub phase-a-selftests job, GitLab therefore has no adjudicating replacement gate at all, so even an arbitrary number of genuine GSA/register failures leaves the pipeline green. Run both gauntlets in the GitLab template before sharing this non-gating classification with it.
Useful? React with 👍 / 👎.
…nly; refresh 2 badges Fix #1 (unblocks the publish pipeline, broken since v0.3.0): the v0.3.0 GSA/register adversarial- review suites (tests/gsa/, tests/gsa-mcp/, tests/gsa-register/, tests/register/) were absent from ci-suite-manifest.json, so CI's "run every committed suite" step treated them as GATING and they exit non-zero on their DOCUMENTED adjudicated false-positives (verified: 7 suites exit 1). Classify them evidence_only — logged, not gating — exactly like the v0.4.0 suites and tests/attacks|ci. The gauntlets remain the gating regression (they fold in each ADJUDICATION.md; 286 + 395, 0 BREAK). The other model-family harnesses (adopt/assurance/diagnostics/... ) currently exit 0; watch/watchdog are slow (~30-60s) but pass. Badges: GSA v0.1-draft -> v0.3-draft (SPEC-CHANGES [0.3-draft]); gauntlet 282/282 -> 395/395 HOLD, 0 BREAK, link -> tests/gauntlet/v0.4.0-battery.js. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nc5c8hPUqe46upP2YBBBGC
… publish blocker) List B (hygiene-scan.js --ci) was hard-failing every release (incl. v0.3.0/v0.3.1) with exit 2 'HYGIENE_HMAC_KEY not set' — the HYGIENE_HMAC_KEY/HYGIENE_DIGESTS secrets were never configured, so it was failing on missing config, NOT on any banned identifier. Gate the step on the secret being present (job-level env + step if): it runs + enforces when the maintainer configures the secrets, and skips (green) otherwise. List A (honest-language) still always gates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nc5c8hPUqe46upP2YBBBGC
…) — #2b run+resume
The emitted manager (managerJsContent) spawns watchdog.js, which writes WATCHDOG-HALT.json
{dead_man_switch:true} at startup and deletes it on clean manager exit. On clean completion the
manager only did watchdogChild.kill() in releaseLock — signalling the guard but killing it before
its cleanup poll fired, so the dead-man file persisted. A back-to-back second `manager.js ci-run`
(the CI "Run + resume the scaffolded project" step) then misread the stale switch as a dead guard
on a halted run and exited 2 (deterministic on Windows, racy on Linux — reproduced 20/20).
Fix: on the exit-0 success path, BETWEEN `log("__done__","complete",0)` and `process.exit(0)`, stop
our own guard and remove the dead-man artifacts (WATCHDOG-HALT.json + .watchdog-hb/.orphan). This
point is UNREACHABLE on any HALT path — every haltNow() throws to the })().catch error handler
before __done__, a blocked-loop kill SIGKILLs the process before __done__, and even a max_log_bytes
halt during the __done__ log throws before the unlink — so it can never erase real halt evidence. A
genuinely halted/incomplete run still leaves the switch and its resume still blocks until
--acknowledge-budget. watchdog.js is UNTOUCHED (SIGKILL-persist / dead-man / orphan semantics intact).
Adversarial trace: confirmed no halt/kill path reaches the success-path unlink.
Regression test: scaffold.js --selftest gains e2e-run-then-immediate-resume-completes-clean.
Verified: 20x repro 0/20 (was 20/20); scaffold.js --selftest 63/0 (incl. D3, e2e-real-run-halts-on-
max-steps-breach, e2e-resume-without-ack-is-refused, e2e-acknowledge-budget-resumes); watchdog.js
--selftest, gate/verify selftests, both gauntlets (286+395, 0 BREAK) green; fresh-scaffold
verifyFrozen green (self-hash auto-updated). Header D6 note added.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nc5c8hPUqe46upP2YBBBGC
…-crash disposition Checkpoint detection treated every *.json in the run dir as a completed-step save point, so runtime state (.watchdog-capability.json, budget-state.json) counted as progress. Consequences: the kill landed before any real step had finished, the PA-9 degenerate-run check compared the wrong number against totalSteps, and the resume check demanded a '"status":"skipped"' line for a pseudo-step named ".watchdog-capability" -- whose normal re-write on resume was then reported as "resume is broken". That is the ubuntu CI failure in the verify job. Checkpoints are now exactly the <step>.json files named in pipeline.json, so there is no naming heuristic to drift from the scaffold. With a genuinely mid-flight kill, the watchdog's armed dead-man switch is in play for the first time, and its disposition is OS-dependent rather than a matter of correctness: on POSIX the guard is orphaned but alive, so it withdraws the switch and leaves an .orphan receipt; on Windows the kill takes the whole process tree, the guard dies with the manager, and the switch legitimately persists -- from disk that is exactly the "guard dead, run unfinished" state D4 exists to catch. The harness now asserts BOTH: the hand-off receipt on POSIX, and on Windows that a plain restart REFUSES to resume past a dead guard before --acknowledge-budget is used. Acknowledging unconditionally would also step over a REAL halt, so the harness instead fails loudly if real halt evidence appears during the crash test, or if a guard that is still alive never disarms. Verified: 10/10 green on Windows and 10/10 on Linux; power-loss, exactly-once, same-run lock, stale-lock and pid-reuse probes all still passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
… move opt-in into the scanner SECURITY. 647f301 hoisted HYGIENE_HMAC_KEY to job-level `env:` so the List B step could skip itself when the secret was unset. A job-level secret is in the environment of EVERY step in that job, including List A (scripts/docs-lint.js), which runs on `pull_request` from the PR's own working tree -- a same-repo PR could edit that script and exfiltrate the key. The job-level `env:` block is removed; the List B secrets are step-scoped again. That was only possible because a workflow can only branch on a secret by exposing it to the job, so the opt-in decision moves into hygiene-scan.js itself (ciConfigStatus): - both secrets set -> enforce, a finding is a red build (unchanged) - neither set -> exit 0, clean skip; List B is opt-in and with no key and no digests there is nothing it could check - exactly one set -> exit 2. Previously key-set + digests-missing ran and hard failed forever, contradicting the documented green skip; silently skipping instead would be worse, because a maintainer who set one secret believes the scan is running. Half-configured must never report clean. List A still always gates, and List B still never runs on `pull_request` at all. Five selftests pin the three-way rule (including GitHub injecting '' for a missing secret) so a later edit cannot quietly turn a half-configured List B green. Verified: hygiene-scan --selftest 32/32; unconfigured -> exit 0; key-only -> exit 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
… the GitLab template too
A count is not an identity. Both batteries compared a suite's FAIL *count* against
the number of adjudicated false-positives in its ADJUDICATION.md, so if a
documented false-positive started passing (a tester harness got fixed) at the same
moment a NEW real failure appeared, the total was unchanged and the battery
reported "hold" -- a live regression hidden behind someone else's adjudication.
Each adjudicated FAIL is now matched BY NAME and consumed at most once, so a
substituted failure is a BREAK. Identity = the FAIL line after "FAIL ", truncated
at the first "{" and whitespace-collapsed; the truncation is only for harnesses
that print the component's own JSON verdict (ids/hashes vary per run). Everything
else in the line -- including "expected=... actual=..." -- is part of the identity
on purpose: if the component's behaviour changes, the adjudication stops matching
and a human re-adjudicates rather than inheriting a verdict reached about
different behaviour. An adjudicated name that no longer fails is reported as STALE
but never gates: a fixed tester assertion is progress, not a regression.
The adjudicated names were taken from the observed FAIL lines and cross-checked
against each per-lane ADJUDICATION.md; the matcher carries its own selftest
(including the substitution case the count gate missed) and that selftest is
folded into both batteries as a gating scenario -- if the matcher stopped
distinguishing identities, every verdict it produced would be worthless.
ci-templates/gitlab-ci.yml now runs both gauntlets. Without them the evidence-only
suites were completely ungated on GitLab: a real regression in any v0.3.0/v0.4.0
control would have passed a green pipeline.
Verified: v0.3.0 287 scenarios / 0 BREAK / 10 adjudicated matched by name;
v0.4.0 396 scenarios / 0 BREAK / 28 matched; no stale adjudications.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…when we stopped it (E) On the clean-completion path the manager set watchdogExpectedExit = true, killed its watchdog and deleted the dead-man artifacts unconditionally. If the guard had ALREADY died (crash, OOM, external kill) with its 'exit' event still queued, that pair both suppressed haltOnDeadWatchdog and deleted the switch -- masking a guard death on an otherwise-completed run, which is the one case where the dead-man switch is the only surviving evidence. The clear is now conditional on the guard actually having been alive, checked against the ChildProcess bookkeeping AND the OS (signal 0), because a death microseconds earlier has not been delivered to this event loop yet and both must agree. If it was already dead, the switch and its evidence are preserved and a loud __watchdog__ line explains why the next start will demand --acknowledge-budget. Unchanged on purpose: the clear still happens only after __done__, so it remains unreachable on every halt path (a blocked-loop kill ends the process group before __done__; a supervisor HALT throws to the error path). A completed run whose guard died at the very end still exits 0 -- every step's checkpoint is durable and the evidence survives -- rather than retro-failing a finished pipeline. Verified: scaffold.js --selftest 63/63 pass, 0 failed (all #2b/D6 cases green). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…rocess group
killProcessGroup() relied on process.kill(-pid, SIGKILL). That only reaches a
process group if the watched process is that group's LEADER. A manager launched
from a shell usually is; one spawned by another process -- a test harness, a
supervisor, an IDE runner -- is NOT. There kill(-pid) raises ESRCH and the old
fallback killed the manager ALONE, leaving its workers running. Orphaned workers
can keep firing external effects after the guard has declared the run halted,
which is precisely what the tree kill exists to prevent. Two independent testers
found it: tests/watchdog/grok ("orphaned children survived kill ...
markerAlive=true") and tests/watchdog/gpt-sol-pro ("process-tree-kill").
The descendant set is now enumerated from the process table (/proc on Linux,
`ps -Ao pid=,ppid=` elsewhere), snapshotted BEFORE the root is killed -- once the
root is gone its children are reparented and the parent links are lost -- and
killed deepest-first so a parent cannot spawn a replacement while we work upward.
The group kill is still attempted first, so nothing that used to work changes.
Windows already walked the real tree via taskkill /T /F and is untouched.
Also fixes the gpt-sol-pro harness's liveness check, which read signal 0 alone.
Signal 0 succeeds for a ZOMBIE -- a process that has been killed but not yet
reaped -- so inside a container whose PID 1 does not reap orphans, a correctly
killed grandchild lingers forever and the harness reported a working tree kill as
failed. It now treats state Z in /proc/<pid>/stat as dead, and falls back to
signal 0 where there is no /proc.
Verified on Linux: watchdog --selftest exit 0; tests/watchdog/grok exit 0
(was FAIL); tests/watchdog/gpt-sol-pro 8/8 PASS (was 7 PASS / 1 FAIL). Both green
on Windows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…on-zero tests/assurance/gpt-sol-pro died on every Linux and macOS CI leg with "FAIL harness.unhandled - TypeError: Cannot read properties of undefined (reading 'length')", which aborted the suite after 4 of its 17 cases. The checkOpaqueReports fixture writes a ~500KB JSON report to stdout and then called process.exit(1). On Windows stdout-to-pipe is synchronous so the whole write landed; on POSIX it is asynchronous and process.exit() discards whatever is still buffered, so ext-tool-runner.js received truncated JSON and opaque_data's fields were undefined. The component was never at fault. Setting process.exitCode instead lets Node drain stdout and still exit non-zero, which is what the case actually asserts. The assertion is also defensive now: every field is existence-checked before its .length is read, so a future truncation reports as a FAIL for this one case instead of an unhandled TypeError that takes the whole suite down with it. Verified on Linux: 17/17 PASS (was 4 PASS / 1 FAIL then abort). Green on Windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…ne downgraded) The Phase A "Run every committed suite" gate was red on all six legs. 24 suites across 13 families (adopt, assurance, diagnostics, evalenv, event-compiler, evolve, graphlint, heal, risk-policy, scaffold, watch, watchdog, watcher) were in neither manifest list, so they ran under the fail-safe rule -- gating, with a warning naming each one. Each was classified BY IDENTITY against the manifest's own rule rather than by whether it happened to be red. All 13 target a real GraphSmith component (scripts/adopt.js, assure.js, diagnostics.js, evalenv.js, event-compiler.js, evolve.js, graphlint.js, heal.js, risk-policy, scaffold.js, watch.js, watchdog.js, watcher.js), so all 13 are per-component REGRESSION suites and are now listed as GATING explicitly. They were already gating; listing them removes 13 warnings and makes the decision reviewable. NONE was moved to evidence_only. Ten of the 13 already passed everywhere. The three that were genuinely failing were FIXED, not reclassified (see the three preceding commits): a POSIX stdout-flush bug in the assurance fixture, the watchdog's process-group kill missing non-leader trees, and -- in a family that was already listed as gating -- the scenario evaluator scoring a correct halt-on-uncertainty as a failure. Reclassifying any of them to make the gate green would have un-gated real components. Verified: the gate passes on Linux (74 discovered, 0 unknown, 0 failing gating suites; 21 evidence-only findings logged and still gated by both gauntlets) and on Windows. Full local Phase A pass -- syntax check, graphlint selftest + dogfood, all eight component selftests, every committed suite, both gauntlets, scaffold + run + resume, and the chaos harness -- is green on both platforms. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…ace as CORRUPT_STATE
_acquireLock created the lock in two steps:
const fd = fs.openSync(this.lockPath, "wx"); // the file now EXISTS...
fs.writeSync(fd, JSON.stringify(record)); // ...but is empty until here
A competing acquirer that hit EEXIST called _readLock() immediately, read "" (or a
truncated record), and the store reported
CORRUPT_STATE: Invalid JSON in state.lock: Unexpected end of JSON input
That is the wrong verdict in the dangerous direction: an ordinary, expected lock
race -- and a crash between creating and writing the lock -- was reported as STATE
CORRUPTION, so a caller that correctly treats CORRUPT_STATE as fatal halts a
perfectly healthy run and tells the operator their state store is damaged. Both
halves of the window were reproduced deterministically (an empty lock file and a
half-written one each produce CORRUPT_STATE), and it fired under load: 3 of 4
stress rounds (5 processes x 600 register/deregister pairs) hit it before this
change, 0 of 4 after.
Two parts:
1. The lock record is written to a temp file, fsync'd, then hard-LINKED into
place. link() fails EEXIST when the lock is held, so the exclusive-create
contract "wx" provided is preserved exactly, while the record becomes visible
atomically and fully formed -- the window no longer exists. Filesystems
without hard links (EPERM/ENOSYS/EXDEV/EOPNOTSUPP/ENOTSUP) fall back to the
historical two-step create, which part 2 covers.
2. In the EEXIST path, a lock that exists but does not parse is now RETRIED
within the existing bounded attempt budget (with a short wait, so the retries
outlast a write window) instead of failing on first sight. If it is still
unreadable after every attempt it IS reported as CORRUPT_STATE, naming the
likely cause and the remedy -- persistence is what distinguishes a lock being
born from a damaged one, and no single observation can. It is never silently
stolen: the steal path requires an owner_token an unparseable lock cannot
supply, so a hostile or garbage lock file cannot be removed by this path.
_readLock() itself is unchanged and still strict, so every other caller keeps
failing closed on genuine corruption. A new selftest case pins both properties:
the lock is created fully formed with no temp file left behind, and an unreadable
lock is retried across the budget before it is condemned.
Verified: state-store --selftest 6/6 pass; 12,000 register/deregister pairs under
5-way contention with zero CORRUPT_STATE (vs 3 of 4 rounds before).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…; deadline-bounded retries
Both state-store suites failed on every windows and macos CI leg. Diagnosed under
2-core contention: the component was right every time and the harnesses' timing
assumptions were not.
Every registry operation takes the store lock, replays the journal and sweeps
lapsed leases BEFORE running -- correct, and the reason these tests were fragile.
Any assertion of the form "this run is STILL registered / still slotted" had to
win a race against its own lease, and load can only break that direction:
- deepseek test 5 ran the whole register / re-register / conflict / heartbeat /
deregister / list sequence -- about 8 lock+fsync round-trips -- under a 40ms
lease, so the run was swept mid-test. The symptom rotated between
"Re-registration should report existing=true", "Run is not registered" and
"Deregistration failed" depending on where the scheduler stole the time.
Expiry now runs on its own store, where only ONE operation precedes the wait
and the assertion is "HAS expired" -- a direction load only helps.
- grok registry.register-sweep-live-trees-journaled used one 50ms-lease store
for both directions. Runs that must stay live now come from a comfortable
lease and only run-expire-me from a short one, both on the SAME root so every
journal and registry assertion still reads one shared set of files.
- grok window.slots-N-plus-1-... asserted CLOSED_PASS across ~10 operations
under an 80ms lease. A lapsed slotted run is swept as `abandoned`, which sets
the FLAG bit, so the close returned CLOSED_FLAGGED. The deliberate abandoned
path still runs, on its own short-lease store with enough headroom that the
run survives long enough to be slotted before it lapses.
The concurrency battery had two separate problems. Its retry budget was
attempt-bounded (80 attempts x <=25ms backoff, ~1.6s) as a proxy for time, and two
hammering processes exhausted it on a loaded runner -- reported as an error when
the store was behaving correctly by refusing under contention. It is now bounded
by a 10s wall-clock deadline, which states the intent directly, and the outer wait
for the workers was raised to 120s so it outlasts the budget it waits on. It also
recorded only a COUNT of unexpected errors, so the failure read "errors a=0 b=1"
and said nothing about what went wrong; it now records the error code. That is
what surfaced the real CORRUPT_STATE lock-race bug fixed in the preceding commit.
Verified on Linux: 5 consecutive cycles of both suites green under 4-way CPU
contention on 2 cores (the load that reproduced every failure above). Green on
Windows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…, not just exit 2 tests/scaffold/deepseek failed on both macos CI legs. Reproduced on Linux under 6-way CPU load: 6 of 6 cycles, reporting [FAIL] resume/REFUSED-BUT-MUTATED -- exit=2 but budget-state.json was modified during refusal The component was right; the test had two coupled problems, both from the same place -- the watchdog dead-man switch after a SIGKILL. 1. The resume that is MEANT TO DO WORK used a plain `runManager(project, "run")` right after SIGKILLing the manager. That start is REFUSED, correctly per D4, whenever the kill takes the guard with it (the whole process tree on Windows) or lands before the orphaned guard has withdrawn its switch (POSIX). So the resume executed nothing: no second external call, no budget trip, `halted` stayed null, and the run still exited 2. It now passes --acknowledge-budget, the documented operator action after a hard kill, which records the extension and resets no counter. Worth noting this was ALSO masking a weaker assertion: the counter-accumulation check had been passing vacuously as "halted-on-unknown, total=1" and now genuinely reaches external_calls_total=2 across the kill. 2. The immutability check treated EXIT 2 as proof of refusal. A start that runs and then trips a budget also exits 2, so when the preceding sequence had not crossed max_external_calls -- which is load-dependent -- the next start did not refuse, it legitimately RAN, and its correct work was reported as mutation-during-refusal (observed: halted null -> max_external_calls, steps_executed 1 -> 2, stderr "HALT (budget)" rather than "Run previously HALTED"). The precondition is now established explicitly before the snapshot, and the refusal is identified by the refusal itself. Two new failure modes are named rather than conflated: NOT-REFUSED (a halted run that accepts a resume without acknowledgement -- a budget bypass) and refusal-precondition. Verified: 5 consecutive cycles green under the 6-way load that failed it 6/6 before; green on Windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…killing the guard watchdog-death-fail-closed failed on both windows CI legs: FAIL watchdog-death-fail-closed: guard died (SIGKILL); blocked target N remained alive and no halt evidence appeared after 480ms The test spawned the watchdog, slept a fixed 60ms, then SIGKILLed it and asserted that halt evidence exists -- the D4 property that a guard's death stays discoverable. But the watchdog arms its dead-man switch as its first act at STARTUP, and node startup is not free: measured on a loaded 2-core box, the switch appears at t+187..211ms. A kill at t+60ms therefore destroyed the guard BEFORE it ever watched anything. A guard that never started correctly leaves no evidence, and the test scored that absence as "the blocked target escaped" -- condemning the component for a premise the test itself never established. It reproduced 5 of 5 cycles under 4-way load on 2 cores, and never on a fast many-core box, which is exactly why it looked windows/macos-specific. The test now waits for the switch to be armed before killing the guard, so the property under test -- what happens when a guard that WAS watching dies -- is actually the property exercised. The assertion is also stricter: the surviving evidence must be the dead_man_switch record, so the pass is earned by the switch persisting rather than by any file happening to exist at that path. Not touched, and worth knowing: budget-just-under (a 205ms heartbeat interval against a 240ms budget) and budget-just-over (elapsed must land in 240..340ms) have only ~35..100ms of headroom. Neither failed across the runs here, so they are left alone, but they are the same shape and a slower runner could trip them. Verified: 5 consecutive cycles PASS=8 FAIL=0 under the 4-way load on 2 cores that failed it 5 of 5 before; green on Windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…killing it (node 18) After the previous round, CI was down to exactly two red legs: Phase A on windows-latest node 18 and macos-latest node 18. Both node 22 legs on those same runners were GREEN, so this was version-sensitive, not load. Reproduced by installing node 18.20.8 and running the gate: tests/watchdog/grok, case A8-watchdog-self-crash-no-manager-notice-channel. Same root cause as the gpt-sol-pro fix in 9d522c9, in a second suite. A8 spawns the watchdog, sleeps a fixed 80ms, then force-kills the guard's tree and checks whether any halt evidence survived. But the watchdog arms its dead-man switch as its first startup act, and node startup is not free -- measured at t+187..211ms on a loaded 2-core box. A kill at t+80ms therefore often destroyed the guard BEFORE it watched anything; a guard that never started leaves no evidence; and the case read that absence as its documented interface gap. Node 18 starts more slowly than node 22, so it lost the 80ms race far more often -- which is exactly why the failure looked node-18-specific. It now waits for the switch to be armed, so killing the guard mid-watch (the point of the case) is what actually happens. The PASS branch's message was also wrong about why it passed. It said "halt still produced (unlikely race)", but on the normal path the surviving file IS the armed dead-man switch -- the manager-detectable channel this case was written to say did not exist. It does now: the switch persists (D4) and scaffold's manager separately watches the guard's own exit and HALTs fail-closed (D3, covered by its selftest). The message now says which of those it observed. The FAIL branch is deliberately untouched: if that evidence ever stops surviving, a blocked run really is left unguarded and this must still say so. Verified on node 18.20.8: 4/4 runs of the suite green, and the FULL suite gate green ("FAILED gating: none") -- then re-run on node 22.22.2, also green, so the fix is not a version trade. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…below the jitter floor) `scripts/verify.js --selftest` runs on ALL SIX Phase A CI legs, which makes it the only candidate found so far that can explain the macos-18 leg rotating into the failing set — every other fix this session was windows-only or load-only. `profileResumableState`'s clean-round-trip fixture built its store with `leaseMs: 40`, then asserted the run was STILL ACTIVE across a SHA-256 hash, a JSON.stringify, and a second full lock-acquire/fsync/journal-read cycle via a fresh createStore (which sweeps expired leases first, correctly). Margin was 0..40ms against this repo's own MEASURED process/IO jitter floor of 187..211ms on a loaded 2-core box -- already below the floor. On expiry the run's slot flips active -> terminal/abandoned in hashAfter only, roundTripMatch goes false, and --selftest reports failed. Note the asymmetry that makes this class of bug one-directional: asserting something HAS expired is safe (load only helps it); asserting something is STILL live is fragile (load can only break it). This was the latter. Checked first whether 40ms was product behaviour or a fixture value: it is fixture-only. state-store.js honours a caller-supplied leaseMs ONLY when GRAPHSMITH_TEST_MODE=1; outside test mode the constructor always uses DEFAULT_LEASE_MS = 30000. No real caller can ever get a 40ms lease, so this is not a product defect. Widened to 5000ms (the convention already used elsewhere) and added an explicit heartbeat renewal immediately before the comparison window. The file's two OTHER leaseMs:40 fixtures were re-checked and deliberately left alone: the corrupt-state one throws during the sweep itself, unconditional on lease timing; the journal-roll-forward one asserts slot PRESENCE, and sweeping terminalizes a slot in place but never removes it. Verified on node 18.20.8 and 22.22.2: verify --selftest 78/78; tests/verify/ deepseek 124/124; tests/verify/grok 52/52; gemini profiles-tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…-reuse lease
Two hazards in the RUN-LOCK / PID-REUSE probes, found by a systematic audit rather
than by waiting for CI to surface them.
1. The lock probe slept a fixed 150ms "let A take the lock" before spawning manager
B, which must then observe A's lock and REFUSE. 150ms is already BELOW this
repo's measured node-startup jitter floor of 187..211ms on a loaded 2-core box,
and manager.js is heavier than the script that was measured (it loads
state-store and tunables and acquires a lock before the lock file exists). So
the margin was already negative: if A had not yet written its lock, B acquired
it cleanly and the probe failed. Now polls for `.runs/<lkId>/.lock` with a 5s
deadline, and if it never appears that is reported as a specific acquireLock()
defect rather than a timing fluke -- so the change adds an assertion instead of
removing one.
2. The pid-reuse probe's Direction B ("a live pid with a FRESH heartbeat must be
refused") ran on leaseMs=1000, ~800-850ms of margin against the same floor.
Same "still fresh" shape, which load can only break. Raised to 5000ms (~23x).
Direction A backdates the lock mtime by 2 * leaseMs so it scales automatically
and is now 10s past expiry; that direction is safe by construction because
delay only makes it MORE clearly expired. Costs no wall-clock time -- neither
direction ever sleeps out the lease.
Verified: 4 chaos runs per node version (18.20.8 and 22.22.2), all exit 0 with
ALL CHECKS PASSED, every probe line green including power-loss, exactly-once,
stale-lock and both pid-reuse directions.
KNOWN, PRE-EXISTING, deliberately not changed here: roughly 1 run in 5 exits 0 via
the documented "SAFETY PASS (halt path)" branch when the SIGKILL lands inside the
demo worker's write-ahead window, and such a run never reaches the lock,
stale-lock or pid-reuse probes. That is correct behaviour (PA-1: a halt there IS
the safety property working) and the exit code is 0 either way, so it does not
gate -- but a green chaos run does not by itself prove those later probes ran.
Worth a follow-up that resolves the intent and continues, rather than a late
change to this harness's control flow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…ssumptions
The last of the load-sensitivity that made Phase A fail on a ROTATING 2 of 6 legs.
Same two shapes as every earlier round: a fixed sleep standing in for a
precondition, and a numeric window too thin to survive a shared runner. In every
case the component was correct and the test's assumption was not; no assertion was
deleted and no security lower bound was moved.
tests/watchdog/gpt-sol-pro:
- process-tree-kill: a fixed 350ms sleep stood in for "the SIGKILL has landed on
the grandchild", which has no ChildProcess handle to await. Now polls for both
pids to actually be gone (5s bound).
- budget-just-under: probed within 35ms of the budget (205ms heartbeat vs 240ms).
The guarantee is "no halt while heartbeats arrive well inside budget", which
does not require probing the edge -- so the STIMULUS was widened (90ms
heartbeat) and the assertion left exactly as it was.
- budget-just-over: acceptance ceiling 100ms -> 1000ms. The LOWER bound
(elapsed_ms > BUDGET) is the security invariant -- never a false-early kill --
and stays exact. Only the "how long may lawful detection take" tolerance moved.
tests/watchdog/grok:
- A3-process-tree-kill-no-orphans: fixed 300ms "let the OS reap" -> bounded poll
on the descendant pids and the marker file.
- A7b: "overly late" tolerance widened; the lower bound cannot flake by
construction because watchdog.js only ever writes halt evidence from inside
`if (elapsedMs > budgetMs)`. Observed elapsed/budget: 352/350, 385/350 -- a
2-35ms live margin against the old ceiling.
- A7a: stimulus widened (budget 600 -> 1800ms against an unchanged 80ms
heartbeat, ~22.5x separation). Also added a missing bounded timeout on
`wd.done`: this case had NONE, so a real defect would have hung the entire
suite instead of failing cleanly.
tests/watch/grok:
- A5 process-group-kill: fixed 800ms settle wait -> poll until manager and child
are actually reaped (10s bound).
- A9 continuous-tail: sleep-and-hope-N-poll-ticks-landed -> wait for the first
frame on stdout, then for the mutated marker to actually appear, before killing
the child; the CLI's own safety-net timeout raised so it cannot race the test.
tests/state-store/deepseek test 7 (concurrency, windows-gated -- maps onto the
windows-18/22 legs):
- Retry budget was 60 attempts x 2-8ms backoff (~120-600ms) used as a PROXY for
wall-clock time, thinner than the sibling grok budget already proven to flake.
Now a 10s deadline per run-id, matching the fix already made in grok.
- It recorded only a COUNT of unexpected errors. A bare count is what made an
earlier failure read "errors a=0 b=1" and hid a genuine CORRUPT_STATE bug for a
whole CI round trip; it now records the error code and surfaces it.
- REAL PRE-EXISTING BUG FIXED: the outer wait resolved a crashed worker to null
and then `.filter(Boolean)` dropped it, so a worker that died without sending
its result vanished from the accounting entirely -- the suite could report PASS
with an entire worker's 10 run-ids never having executed, silently not checking
the "every run-id eventually completes" guarantee it exists to check. Crashed
workers are now surfaced as an explicit error result and the result count is
asserted, with a bounded outer deadline that outlasts the workers' own budget.
tests/watcher/sonnet was audited and needed no changes: fully synchronous, no
sleeps, no duration windows.
Verified on node 18.20.8 AND 22.22.2 (CI uses both): syntax check, graphlint
selftest + dogfood, ten component selftests, the full 74-suite gate with
unknown=0, both gauntlet batteries, scaffold + run + resume, and chaos x3 --
40 checks per version, zero nonzero exits. Same full pass green on Windows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…ne-liner
The "Run every committed suite" step was a ~2000-character `node -e` in
.github/workflows/ci.yml, duplicated as a 3122-character copy in
ci-templates/gitlab-ci.yml. That form had three problems, all of which cost real
time on this PR:
- it could not be run locally the way CI runs it, so the gate itself was never
under test;
- it was invisible to the repo's own `node --check scripts/*.js` step, and one
quoting slip took out all six matrix legs at once;
- when it failed, the suite names were buried ~3300 lines deep in a log viewer
that repeatedly 500'd, froze, and on at least one occasion spliced lines
between two adjacent blocks. Reading that log cost more time than any of the
fixes on this branch.
The classification policy still lives entirely in ci-suite-manifest.json and the
decision is unchanged: gating_suites must be green, evidence_only suites are
adversarial-review harnesses that exit non-zero by design and are logged without
gating, and anything in NEITHER list is treated as gating (fail-safe) with a
warning naming it. Verified byte-identical to the code it replaces: same exit
code, same discovered set, same classification for all 74 suites, and the same
gating / evidence-only / unknown failure sets, on node 18 and node 22.
Two things are new, both additive:
- on a gating failure the runner appends the failing suite names to
$GITHUB_STEP_SUMMARY and re-runs each failure once, labelling it FLAKY
(re-run passed) or reproducible. The exit code does not change -- a
first-run gating failure still fails the job, deliberately;
- validatePrefixList() rejects a manifest whose prefix lists are not arrays of
non-empty strings. An empty-string entry in evidence_only would have matched
every suite path (because "".startsWith("") holds and evidence is checked
first), silently defanging the entire gate from a one-character typo, with
none of the fail-safe's warnings. It now refuses to run instead.
Both CI surfaces now call the same script, so GitHub Actions and the GitLab
template can no longer drift apart.
Fail-safe behaviour re-confirmed on synthetic manifests: gating failure -> exit
1; evidence-only failure alone -> exit 0; suite in neither list -> exit 1 plus a
warning; empty manifest, missing key and malformed JSON all fail closed.
Recorded but deliberately unchanged: if an evidence_only prefix is a
string-prefix of a gating_suites prefix, the shorter evidence entry wins. No such
collision exists in the manifest today; this behaviour is inherited unchanged
from the code being replaced and is called out here so the next prefix addition
gets a human eye.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…lure tests/verify/deepseek/run-tests.js, case `E6. selftest/pass`, wrapped `node scripts/verify.js --selftest` in an execSync with `timeout: 30000`. --selftest runs a 78-check attack corpus over multiple ephemeral fixture builds and is by far the heaviest single operation in the suite: measured 8.7s idle and up to 26.9s under contention on a 2-core box, i.e. as little as ~10% headroom. Every other execSync in the same file measures 217ms-1.1s against a 10s budget. When execSync throws ETIMEDOUT the catch block reports E6 as false, so a harness that gave up waiting is indistinguishable from the product actually failing a check. Reproduced exactly by forcing `timeout: 50` locally: verify.js --selftest still completes and still reports pass, and the case still goes red. A GitHub macOS runner is the slowest leg in the matrix and node 18 starts slower than node 22, which is consistent with this suite being one of the two genuine gating failures on the macos-18 leg while passing everywhere else. Only the wall-clock the harness allows the child is widened, 30000 -> 120000. The assertion is untouched: the output must still parse and all 78 checks must still pass. It still fails if --selftest hangs, crashes, or reports a failed check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…was a no-op
listDescendants() in tests/watch/grok and listDescendantPids() in
tests/watchdog/grok each had two platform branches, and BOTH were broken on a CI
leg:
- POSIX ran `ps -o pid= --ppid <pid>`. `--ppid` is a GNU/procps long option;
BSD ps (macOS) rejects it outright.
- Windows ran `wmic process where (ParentProcessId=...)`. wmic is deprecated
and is no longer present on current Windows -- removed in Windows 11 24H2
and Server 2025, which is what `windows-latest` now is. Confirmed absent
locally: "'wmic' is not recognized as an internal or external command".
In both cases the call threw, the enclosing catch swallowed it, an empty set came
back, and the orphan-detection loop downstream became a no-op that could only
ever conclude "no orphans". A5-process-group-kill-no-orphans has therefore not
actually been checking for orphaned grandchildren on macOS or on current Windows.
It looked green precisely because it had stopped looking.
Both now read the whole pid -> ppid table once and walk it in JS:
`ps -Ao pid=,ppid=` on POSIX (only -A and -o keyword=, both supported by procps
and BSD ps -- the same form scripts/watchdog.js already uses) and
Get-CimInstance Win32_Process on Windows (the documented wmic replacement).
Failure is no longer silent: if the table cannot be read, or comes back empty, a
warning naming the reason goes to stderr, so a platform where enumeration is
impossible says so instead of quietly passing.
This strengthens the check rather than relaxing it -- it restores coverage that
had been silently lost. Verified: 13/13 on Linux node 18 and 22, and 13/13 on
Windows with the enumeration confirmed working (no degraded warning emitted), so
the now-live orphan assertion finds no orphans to report.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…loor
Four fixtures in tests/state-store/{deepseek,grok}/run-tests.js still configured
leaseMs 40 or 80. The measured floor for node startup plus a first file write on
a loaded 2-core box in this repo is 187-211ms, and on Windows and macOS the
filesystem syscalls are slower again.
Each of the four is immediately followed by a "this lock is STILL live, so it
must be refused" assertion -- the one direction load can only break, per the
governing asymmetry: an assertion that something HAS expired is safe under load,
an assertion that something is STILL live is not. With a 40ms lease the lock
could look expired before the refusal check even ran.
Affected cases:
lock.pid-alive-stale-steal-fresh-refuse-env-gate (grok)
attackLockStealAndTokenMismatch's "fresh lock not refused while held" (grok)
test1_lockStealMismatch's "fresh lock by alive pid was not refused" (deepseek)
test2_pidReuseAndTestMode's "fresh heartbeat from alive pid was not refused"
Raised to leaseMs 2000, heartbeatMs 200. Every paired steal-when-STALE assertion
in these same functions establishes staleness with an explicit 5000ms utimesSync
backdate rather than by waiting out the lease, so it stays correct for any lease
well under 5000ms -- both halves of each test still assert what they did before.
Same fix class as f00cd0e and 4e7ea81; these four were the remainder.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…heck trips harness.no-real-tree-mutation in tests/assurance/gpt-sol-pro/run-tests.js hashes every file under the repo root before and after the suite's temp-driven CLI attacks, and on a mismatch reported only "repository snapshot changed while the harness was running". It named no file. This suite is one of the two genuine gating failures on the macos-18 leg. It passes on Linux node 18 and 22 in isolation, in the full battery, under CPU contention, and with a symlinked long TMPDIR; the macOS-tmpdir-realpath and macOS-16KB-pipe-buffer hypotheses were both tested with real simulations on Linux and both came back clean, as did the theory that a process from an adjacent suite outlives it and writes into the tree during the snapshot window. Which leaves a check that fails on a platform we cannot reproduce and refuses to say why. snapshotDelta() now enumerates the added, removed and changed relative paths (up to 12, then a count). What counts as a violation is unchanged -- snapshotsEqual still decides pass/fail -- so this weakens nothing; it means the next red run on that leg identifies the file instead of restating that a file exists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
This is the root cause of the macos-18 leg, and it is one bug behind all four
failing cases.
`process.stdout.write()` to a PIPE is asynchronous on POSIX. libuv performs one
write(2) and queues whatever did not fit; `process.exit()` then discards that
queue. The caller reads a truncated report next to exit code 0 -- silent, and
indistinguishable from the product genuinely failing a check.
A Linux pipe holds 65536 bytes, so a ~12 KB report always lands in a single
syscall and nothing is ever queued. That is the only reason this has been green
on Linux for the life of the branch. On the macos-latest node-18 leg it is not:
tests/verify/deepseek E6. selftest/pass
"verify --selftest failed: Unexpected end of JSON input"
tests/assurance/gpt-sol-pro assure.minimal-packet
assure.byo-contained
assure.honest-floor
Those three assurance cases are the only three that parse the `assure.js`
report; the other fourteen cases in that suite passed. Not three defects -- one
report that arrived incomplete.
Reproduced on Linux, before and after, by shrinking a real pipe with
fcntl(F_SETPIPE_SZ) and draining it concurrently exactly as execSync does:
verify.js --selftest 11712 bytes of report
pre-fix pipe=4096 exit=0 delivered 4096 JSON broken
post-fix pipe=4096 exit=0 delivered 11712 JSON parses
both pipe=16384/65536 delivered 11712 JSON parses
assure.js <project> --byo <byo> --no-scenario 11998 bytes of report
pre-fix pipe=4096 exit=1 delivered 4096 JSON broken
post-fix pipe=4096 exit=1 delivered 11998 JSON parses
The fix writes the report to fd 1 with fs.writeSync in a loop, so every byte is
handed to the kernel before the function returns and nothing can be sitting in a
queue when the process exits. Exit codes and exit timing are unchanged.
Chose this over `process.exit(N)` -> `process.exitCode = N`. That also fixes the
truncation, but it lets the process keep running until the event loop drains,
which trades a truncated report for a possible hang if any handle is still open
-- a different failure mode, in a CLI whose whole job is to exit with a verdict.
fs.writeSync keeps the explicit exit and simply guarantees the bytes are gone
first.
Scope is bounded by evidence, not by pattern-matching. Every other report-
emitting script in scripts/ either emits under 4096 bytes or has no test that
parses its stdout (checked: scaffold.js emits the largest report in the repo at
23 KB from --selftest, but no suite parses it, and the suites that do invoke it
pass it a project name and read only its exit code). Those exits were left
alone. The same helper is available if that ever changes.
Verified green after the fix, on node 18 and node 22:
verify.js --selftest 78/78; assure.js --selftest 7 checks
tests/verify/deepseek 124 pass / 0 fail (123 on Windows, one platform-gated)
tests/assurance/gpt-sol-pro 17 pass / 0 fail
full suite battery: 74 discovered, 0 gating failures, 21 evidence-only
v0.3.0 gauntlet 287 scenarios all HOLD 0 BREAK
v0.4.0 gauntlet 396 scenarios all HOLD 0 BREAK
graphlint scripts/ clean
Same battery, both suites, both selftests and graphlint also green on Windows.
Found because the previous commit made the suite gate write its failures to
$GITHUB_STEP_SUMMARY. The FAIL identities above came from that summary; the log
viewer had never made them readable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…s on PATH Nine execSync call sites across three suites invoked a bare `node`: tests/verify/deepseek/run-tests.js 6 sites (lines 610, 719, 859, 912, 977, 1174) tests/graphlint/gemini/run-tests.js 2 sites tests/adopt/glm/run-tests.js 1 site Two problems, one of them not obvious. The visible one: on any machine where node is not on PATH the suite dies mid-run with `/bin/sh: node: command not found`, status 127. Found by running the suite on a real macOS box with node installed outside PATH -- tests/verify/deepseek aborted before it ever reached the case we were trying to observe. The quiet one: nothing guarantees that `node` on PATH is the same interpreter running the suite. CI happens to be fine because actions/setup-node puts the matrix version first, but a matrix whose entire purpose is to exercise node 18 and node 22 should not depend on that coincidence -- a stray node earlier on PATH would have the suite silently testing a third version while reporting the matrix one. process.execPath is the interpreter we are demonstrably running under. Quoted, so paths with spaces are safe. No assertion changed. Verified: adopt/glm, graphlint/gemini and verify/deepseek all green on Windows (verify: 123 pass / 0 fail) and on Linux node 18 and node 22, plus verify/deepseek 124/124 on real macOS 12.7.3 with node 18.20.8. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…th a control New GATING suite: tests/report-integrity/harness/. It closes the class of defect that aad364e fixed two instances of, rather than leaving the rest latent. THE DEFECT CLASS process.stdout.write() to a pipe is asynchronous on POSIX: libuv attempts one write(2) and queues the remainder; process.exit() discards the queue. The caller reads a short -- or entirely empty -- report next to exit code 0, indistinguishable from the product genuinely reporting success. In a tool whose output IS the evidence, that is the worst available failure mode. Now measured on real hardware rather than inferred. MacBook Air (2015), macOS 12.7.3, node 18.20.8, at the commit before the fix: verify.js --selftest file=11886B pipe=8190B -- short by 3696B, exit 0 and with the fix: 11886B both ways. The effective pipe capacity there is ~8 KB, not the 16 KB I had assumed, and the loss only appears through spawnSync -- piping to a continuously-draining reader never truncates, which is why an earlier `| cat` measurement wrongly looked like a clean bill of health. Two properties of this defect make it unusually good at hiding, both now encoded: - It is NOT size-bounded. In CI the report arrived completely empty, meaning libuv's first try_write moved zero bytes. A small report is not safe, so the suite checks every CLI whose report a gating suite parses, not just large ones. - It gets HARDER to see under load. The pre-fix assurance suite passed 17/0 under 8-way CPU contention and failed 3/3 idle: contention shifts the reader/writer interleaving in the harness's favour. "Run it under load" would have hidden this, not found it. WHAT THE SUITE ASSERTS For each of 21 (script, args) pairs -- every CLI whose JSON report is JSON.parse()d by a suite classified gating in ci-suite-manifest.json -- capture stdout through spawnSync (the asynchronous pipe path the failing suites use) and require it to be byte-identical to the same command redirected to a file (the synchronous path, i.e. the report the author intended). Repeated 3x, because this is an interleaving race and one green run proves nothing. Exit code is deliberately NOT part of the pass condition: the exit code was 0 throughout. A CLI that emits nothing on stdout is reported as out of scope. A CLI that emits output this suite cannot parse as a JSON document OR as JSON Lines FAILS rather than being skipped -- an unrecognised shape means either the report changed or the recogniser needs extending, and silently passing on it is how a check stops checking. Recognising only single documents had already downgraded watchdog.js and loaders.js to "skipped" before that was corrected. THE POSITIVE CONTROL fixtures/leaky-report.js commits the anti-pattern deliberately with a ~256 KB payload, and the suite requires it to LOSE data before trusting any green result above it. Without that, "all reports intact" is ambiguous: it could mean the CLIs are correct, or that the platform cannot exhibit the defect and every check is inert. Observed behaviour: Linux node 18 control lost 183703B of 402967B -> detector live, 21/21 checked Windows control survived intact -> detector INERT, stated plainly Windows pipe writes are synchronous, so inert is correct there -- and the suite says so in its output instead of banking a green that means nothing. That line is the point: a leg where this cannot fail is a leg with no coverage, and it should say so. DO NOT "fix" fixtures/leaky-report.js. Its brokenness is load-bearing. KNOWN GAP, stated rather than hidden: assure.js is only exercised via --selftest, whose report is 2117B and survives on every platform tested. The invocation that actually broke was the ~12KB packet from a real project run. So this suite would NOT presently catch a regression in assure.js. Closing that needs a project fixture here, or a source-level invariant; it is the next piece of work, not something this commit claims to have done. Listed explicitly in ci-suite-manifest.json as gating. Runs ~2 min on Linux. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
Closes the class that aad364e fixed two instances of. 30 emit sites across 17 scripts now hand their report to the kernel before the process can exit. WHY THE EARLIER SCOPE WAS WRONG aad364e fixed verify.js and assure.js and argued the rest were safe because their reports were under 4096 bytes. That reasoning does not survive the evidence: - The defect is not size-bounded. On the macos-18 leg the report arrived COMPLETELY EMPTY -- libuv's first try_write moved zero bytes, so the whole report sat in a queue that process.exit() discarded. Payload size never entered into it. - The audit filter was also wrong. It only looked for process.stdout.write, so every script emitting via console.log was misclassified "ok". console.log goes through the same asynchronous stream. loaders.js, state-store.js, capability-policy.js, promote.js and heal.js were all in that blind spot. Re-audited on the criterion that actually matters -- does a GATING suite parse this CLI's stdout -- and found 33 real emit sites across 18 scripts, of which 30 across 17 are fixed here. MEASURED, NOT ARGUED MacBook Air (2015), macOS 12.7.3, node 18.20.8, verify.js --selftest captured through spawnSync: before 8190 bytes of an 11886-byte report, JSON unparseable, exit 0 (3/3 runs) after 11886 bytes, parses (3/3 runs) TWO SCRIPTS ARE DELIBERATE EXCEPTIONS scripts/watchdog.js keeps an INLINE copy of the helper rather than requiring the shared module. scaffold.js copies watchdog.js -- and only watchdog.js -- as a standalone file into every generated project. A local require resolves fine at scaffold time and then dies with MODULE_NOT_FOUND inside the generated project, taking sync-execution budget enforcement down with it. tests/scaffold/deepseek and tests/scaffold/gpt-sol-pro caught exactly that (halt=tripwire/watchdog-guard-died, "pid ..., code 1"), which is how the constraint got documented in the file. scripts/scenario.js is NOT touched here even though it has two affected sites. It is the evaluator file listed in ci-check-pr-separation.js's EVALUATOR_EXACT, so changing it belongs with the evaluator PR, not with a behavior change. Its two sites travel with fix/scenario-halt-certification. ONE REGISTRATION POINT, ON PURPOSE diagnostics.js self-scans its own requires against ALLOWED_REQUIRE_MODULES and its selftest failed with unlisted-require ./write-report.js until it was recorded there. That allowlist exists so acquiring a dependency is a conscious act; the entry is added explicitly rather than the scan being relaxed. ALSO CORRECTED: A DEFECT IN THE NEW GATE ITSELF tests/report-integrity compared the pipe capture byte-for-byte against a file-redirected run. That is wrong for reports carrying a pid, an elapsed_ms or a timestamp -- watchdog.js is one, and the suite duly failed with file=4726B vs pipe=4731B, a difference in the wrong direction. A flaky gate gets ignored, which is worse than no gate. Determinism is now MEASURED, not assumed: two file-path runs decide it. Where the report is byte-stable the suite keeps the strongest check (byte-identical). Where it is not, it requires the capture to parse AND the JSON-Lines document count to match -- the count matters because a JSONL report clipped exactly at a line boundary would otherwise still parse. Confirmed on real macOS that the weaker path still catches the real defect: 8190 bytes of the 11886-byte report is unparseable, 3/3 runs. verify.js and assure.js now require the shared module instead of each carrying a duplicate, so there is exactly one definition of the guarantee (plus watchdog.js's documented standalone copy). VERIFICATION Linux node 18 and node 22: full battery 75 discovered, 0 gating failures, 21 evidence-only v0.3.0 gauntlet 287 scenarios, all HOLD, 0 BREAK v0.4.0 gauntlet 396 scenarios, all HOLD, 0 BREAK report-integrity 22/22, positive control LIVE (lost 183703B of 402967B) Windows: every scripts/*.js syntax-checks; graphlint clean; 20 component selftests pass report-integrity 22/22, control correctly reports INERT (Windows pipe writes are synchronous, so that leg honestly declares it provides no coverage) scaffold + run + resume + chaos all exit 0 -- a generated project still works Real macOS 12.7.3, node 18.20.8: before/after on verify.js as above; tests/verify/deepseek 124/124 and tests/assurance/gpt-sol-pro 17/0 on the fixed tree Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…dlines With gh authenticated it was finally possible to read the failing job logs directly. Both remaining red legs had a named cause, and one of them dissolves a mystery that has been in the notes as "unexplained" for most of this branch. THE PHANTOM CLASSIFICATION BUG NEVER EXISTED The evidence-only list was printed with console.log and the FAILED-gating list with console.error. In a captured CI log those two streams interleave unpredictably, so evidence-only suites appeared UNDER the "FAILED gating suite(s):" header. That is what produced a macOS log reading as though tests/v040/** -- listed plainly as evidence_only in ci-suite-manifest.json -- had been classified as gating on macOS but not on Linux. Hours went into hunting that, including running the full battery against two trees (main, and main + e50f284) to compare classification behaviour, and writing up a conclusion of "not reproducible at this HEAD; either the paste is not from this commit or the log view is stitched". The earlier observation that the ordering was impossible for a single loop was correct; the reason was two streams, not a stitched log. Verified from the run #67 log: on macos-22 there were 21 evidence-only failures and exactly ONE gating failure. Every tests/v040/** entry classified evidence_only, exactly as the manifest says. No inconsistency ever occurred. All gating-failure reporting now goes to stdout. The runner emits zero bytes on stderr, so the interleaving is structurally impossible rather than merely unlikely. Exit codes and the step summary are unchanged. TWO REAL FAILURES, BOTH THE SAME SHAPE Both are process-lifecycle assertions whose deadlines were too tight for the slowest runners, and both re-ran clean -- the runner's own FLAKY classifier said so, which is what it was built for. tests/watchdog/gpt-sol-pro process-tree-kill red on ubuntu-18 (run #62): parentAlive=false childAlive=true. The tree kill had simply not finished landing on the grandchild. SIGKILL delivery plus kernel process-table teardown for a deep descendant is not free. waitClose(watchdog) 5000 -> 20000, waitClose(parent) 1500 -> 15000, waitFor(both gone) 5000 -> 30000. Only this case is touched; the other waitClose(watchdog, 5000) sites in the file have no failure evidence. tests/watchdog/grok A8-watchdog-self-crash-no-manager-notice-channel red on macos-22 (run #67), green on macos-18 and green on the IDENTICAL SHA in run #66. The case waits for the dead-man switch to arm before killing the guard mid-watch; 10000ms was not enough on a loaded macOS runner. Arm wait 10000 -> 45000. Every one of these waits for something that HAS to have happened, so a longer bound cannot mask a real failure: a tree kill that never lands still reports parentAlive/leafAlive, and a guard that never arms still fails below. A8 ALSO REPORTED A FALSE ARCHITECTURAL FINDING On timeout, A8 recorded: "watchdog death leaves blocked run unguarded; no interface for manager to detect guard death (no reverse heartbeat / exit pipe requirement)" That is a claim about a PRODUCT GAP, emitted when the harness had merely run out of time waiting for the guard to arm. The dead-man switch does exist and does work -- it is what several commits on this branch built. A timeout dressed up as a design finding is exactly the dishonesty this suite exists to catch, so the two cases are now distinguished: an unarmed guard reports INCONCLUSIVE (harness) and names the timeout, and the interface-gap verdict is only claimed when the guard demonstrably armed and the death still went unnoticed. ALSO: REMOVED AN UNSOUND CHECK FROM THE NEW GATE tests/report-integrity compared the pipe capture byte-for-byte against a file-redirected run, deciding "is this report byte-stable" by running the file path twice. That does not work: a low-resolution timing field can match across two runs and differ on the third, so the suite declares determinism and then fails on a legitimate variation. False positives in a gating suite are how gates come to be ignored. Byte-equality also bought no detection power, because every loss this suite exists to catch destroys the report's structure. What remains: the capture must parse (as one document or as JSON Lines) and the document count must match the file path -- the count because a JSON-Lines report clipped exactly at a line boundary would otherwise still parse. Confirmed on real macOS that the remaining checks still catch the real defect: 8190 bytes of an 11886-byte report is unparseable, 3/3 runs. Three consecutive full runs of the suite now pass 22/22. Note for the record: this change is correct on its own merits but it was NOT the cause of the macos-22 red. The report-integrity suite passed on that leg. VERIFICATION Linux node 18: battery 75 discovered, 0 gating failures, 21 evidence-only; v0.3.0 gauntlet 287 scenarios all HOLD 0 BREAK; v0.4.0 gauntlet 396 all HOLD 0 BREAK; graphlint clean; both watchdog suites exit 0 with zero FAIL lines; report-integrity 22/22 on three consecutive runs; runner still exits 1 on a gating failure with 0 bytes on stderr and the step summary still written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…n while blind
A8-watchdog-self-crash-no-manager-notice-channel reported
"watchdog death leaves blocked run unguarded; no interface for manager to
detect guard death (no reverse heartbeat / exit pipe requirement)"
when the harness had merely run out of time waiting for the guard to arm. That is
a claim about the PRODUCT emitted from evidence that did not support it. Auditing
for the same shape found something worse pointing the other way.
WHY A TEST LIES: THE VERDICT VOCABULARY LEAVES NO HONEST OPTION
A check whose preconditions never held has not passed and has not found a defect:
it did not run. PASS claims an untested guarantee. FAIL asserts an unobserved
defect. SKIP means "not applicable here", a different untruth. A test in that
position has no honest verdict available, so it emits a dishonest one. Adding the
missing verdict is the fix; patching individual messages is not.
INCONCLUSIVE is a convention on the output line, not a framework change -- there
is no shared test framework here, 75 suites carry hand-rolled harnesses (rec,
record, report, [PASS]), and unifying them is not worth it:
FAIL <name> - INCONCLUSIVE (harness): <reason>
It counts as a FAILURE so the gate stays fail-closed. A check that never executed
must never read as green -- the same rule ci-suite-manifest.json already applies to
unlisted suites, and the same rule PA-1 applies to the product itself. But it is
never counted or displayed as a product finding: ci-run-suites.js detects the tag,
says so inline on the re-run, and gives it a separate step-summary section stating
in words that nothing in it speaks to product correctness, plus the standing rule
that a recurring inconclusive is a bug with an owner -- a deadline too tight or a
component too slow to arm -- and never a standing exemption.
THE WORSE FINDING: A GATING SUITE THAT PASSED WHEN THE DEAD-MAN SWITCH DID NOTHING
tests/scaffold/deepseek had:
if (haltData && haltData.halt === true) {
pass("watchdog/halt-file-written", ...);
} else {
// Watchdog may not have fired - could be a Windows/scheduling issue
pass("watchdog/halt-not-written", "watchdog did not fire within test window");
}
BOTH branches called pass(), with a comment excusing the miss in advance. So the
dead-man switch -- the most safety-critical mechanism in the product, and the thing
this branch exists to protect -- could fail to arm entirely and this GATING suite
reported success and exited 0. Downstream, halt-resume-test then recorded SKIP
("not applicable", untrue: its precondition simply never materialised) and the
--acknowledge-budget check vanished with no verdict at all, so its absence was
invisible in the counts.
All three now report INCONCLUSIVE naming exactly what did not happen.
PROVEN WITH A POSITIVE CONTROL, NOT ASSERTED
In a scratch copy, muted the watchdog's dead-man switch write so the guard runs,
kills, and never arms:
before this commit exit 0 TOTAL=29 PASS=28 FAIL=0 <- fully green
after exit 1 TOTAL=29 PASS=25 FAIL=3 <- all 3 INCONCLUSIVE
The control also discriminates rather than failing on everything: muting only the
budget-evidence write, leaving the dead-man switch intact, still passes 28/0. And
removing scripts/watchdog.js entirely produces 3 legitimate SKIPs, because the
product feature-detects its absence -- that IS "not applicable", and SKIP is right
there.
TWO MORE INSTANCES OF THE SAME BLINDNESS
tests/watchdog/grok A3-process-tree-kill-no-orphans initialised childPids to []
and swallowed the read error, so an unreadable child-pids.json made the orphan
list empty, which read as "no orphans survived" -- a PASS asserting the guarantee
at the exact moment the harness had no idea which pids to look for. This is the
same defect as the ps/wmic enumeration that had silently disabled orphan detection
on two platforms. childPids now starts as null and an unreadable list is
INCONCLUSIVE. Never having learned the pids is not evidence that there are none.
The tree-kill verdict "manager survived; cannot judge tree kill" was already
phrased honestly -- it declines to conclude -- but was filed as a product FAIL, so
an unjudgeable trial was indistinguishable from a failed tree kill in every count
and summary. Now tagged.
AUDIT NUMBERS, AND WHAT I DID NOT TOUCH
Scanned all 75 suites: 170 empty `catch {}`, of which 113 are benign cleanup
(try { unlink } catch {} is idiomatic) and 3 swallow a precondition. 5 FAIL
verdicts assert a product property with no observed values interpolated.
Three of those 5 turn out to be honest on inspection and were left alone:
risk-policy/glm's two "pattern missing from policy" verdicts carry the pattern id
in the case name and describe something actually read from the policy, and
watchdog/grok:413 already passes the observed values as a fourth argument. Editing
correct code to hit an audit number would be the same dishonesty in a different
direction.
One note on tooling: `node --check` passed a file that referenced an undefined
function. It is a parser, not a linker -- worth remembering before trusting it as
a gate.
VERIFICATION
Linux node 18: battery 75 discovered, 0 gating failures, 21 evidence-only;
v0.3.0 gauntlet 287 scenarios all HOLD 0 BREAK; v0.4.0 gauntlet 396 all HOLD
0 BREAK; graphlint clean; tests/watchdog/grok and tests/scaffold/deepseek both
exit 0 with zero inconclusive lines on a healthy box (the paths only fire when
something genuinely cannot arm, which the positive control above demonstrates).
Runner: synthetic INCONCLUSIVE suite -> exit 1, 0 bytes on stderr, inline note
emitted, dedicated summary section written.
Still to come: GRAPHSMITH_DEADLINE_SCALE and the starvation sweep, which asserts
that under near-zero deadlines every failure comes back INCONCLUSIVE -- a
mechanical detector for this class rather than a patch for its instances.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
…st verdicts
A mechanical detector for the class of defect A8 belonged to, rather than another
patch for one of its instances. It found two more on its first run.
THE IDEA
A test that waits for a precondition and then draws a conclusion can reach the
same verdict two ways: the thing genuinely did not happen, or the wait ran out.
Those are different facts, and a harness that cannot tell them apart will
eventually report the second as the first.
So: scale every HARNESS deadline toward zero and re-run. Under near-zero deadlines
every precondition wait must fail, which leaves an honest suite exactly one
available verdict for those cases -- INCONCLUSIVE. A case that instead emits a
confident verdict is lying by construction. No judgement call, no reading of
intent, just arithmetic.
tests/_harness/deadline.js exposes harnessDeadline(ms), driven by
GRAPHSMITH_DEADLINE_SCALE. Unset or invalid leaves every deadline exactly as
written, so normal runs and CI are unaffected by default -- verified: all three
target suites exit 0 with zero inconclusive lines when the knob is unset.
THE LINE IT MUST NOT CROSS
This scales HARNESS deadlines only -- how long the TEST is willing to wait. It
never scales a PRODUCT budget: a watchdog budget_ms, a lease TTL, a
max_wall_time_ms. Those are the values under test, and scaling them would not
stress the harness, it would change the experiment. The rule is stated in the
module header because it is the one way this tool could quietly become useless.
WHAT IT FOUND IMMEDIATELY
Two catch-all handlers were reporting a precondition timeout as an ordinary
failure, which files a harness problem in the product-findings bucket:
tests/watchdog/gpt-sol-pro FAIL harness-unexpected-error:
Error: timeout waiting for blocked target readiness
tests/scaffold/deepseek [FAIL] harness/internal --
Error: timeout waiting for .../.runs/run/ready
Both now split on the actual error rather than blanket-tagging: "timeout waiting
for X" means the trial never started and is INCONCLUSIVE, while any other
exception is a genuine failure and stays one. Blanket-tagging would have hidden
real product crashes behind a harness label -- the same mistake in the opposite
direction.
PER-SUITE CONTROL, SO THE SWEEP CANNOT PASS VACUOUSLY
A target that passes cleanly under starvation has told us nothing: either it has
no precondition waits, or harnessDeadline() is not wired into it. Reporting that as
a pass would be exactly the blindness being hunted, so each target must show at
least one starvation-induced failure or it is reported NOT COVERED. That is what
caught the wiring gaps while this was being built.
COVERAGE, STATED PLAINLY
Wired and swept: tests/watchdog/grok (including A8's arm wait and A3's reap wait),
tests/watchdog/gpt-sol-pro (waitFor and waitClose), tests/scaffold/deepseek
(waitForFile). These are the timing-dominated suites where every CI flake on this
branch actually lived.
NOT yet wired: tests/watch/grok and tests/scaffold/gpt-sol-pro. Their deadlines are
inline `Date.now() + N` expressions rather than routed through a helper, so each
call site needs converting individually. Claiming corpus-wide coverage today would
be the same overreach A8 made, so the target list is explicit and short, and adding
a suite without wiring it surfaces as NOT COVERED rather than as a pass.
THE MIRROR IS THE SAME KNOB
GRAPHSMITH_DEADLINE_SCALE=5 inflates deadlines instead. Any case that flips from
FAIL to PASS under inflation was timing-limited, not product-limited. That is the
question -- "is this finding real, or is my harness impatient" -- that consumed most
of the debugging on this branch, and it is now one env var away from an answer
across every wired suite.
A8 ITSELF, VERIFIED UNDER STARVATION
[FAIL] A8-watchdog-self-crash-no-manager-notice-channel — INCONCLUSIVE (harness):
the watchdog never armed within 45ms, so it was killed before it began
watching -- this says nothing about whether a guard-death notice channel exists.
Note "45ms", not "45000ms": the message reports the budget actually used. An
inaccurate number inside an honesty message would be its own small lie.
VERIFICATION
Linux node 18: battery 76 discovered, 0 gating failures, 21 evidence-only;
harness-honesty 3/3 in-battery; v0.3.0 gauntlet 287 scenarios all HOLD 0 BREAK;
v0.4.0 gauntlet 396 all HOLD 0 BREAK; graphlint clean; all three target suites
exit 0 with zero inconclusive lines when the knob is unset.
Listed in ci-suite-manifest.json as gating.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARvYLenCrqcDcr8VyefoxG
ALSO: A8 RENAMED, BECAUSE ITS NAME WAS THE LAST STALE CLAIM
A8-watchdog-self-crash-no-manager-notice-channel
-> A8-guard-death-leaves-discoverable-evidence-D4
The old name asserted an ABSENCE -- no reverse heartbeat, no exit pipe, no way for
the manager to notice its guard had died -- and that was a true adversarial finding
when the case was written. The dead-man switch has since been built, and the case
now verifies the opposite. Its own passing message already said so:
[PASS] ... guard killed mid-watch; its armed dead-man switch survived, so the
guard's death is discoverable (D4)
So the name contradicted the behaviour it verifies, and anyone reading a green run
saw a case announcing a gap that no longer exists. A name is a claim like any
other, and a stale one is a stale claim.
Safe to rename: it is not adjudicated by name in any ADJUDICATION.md and not
referenced by either gauntlet battery, so no identity matching breaks. The
historical trace is preserved in a comment at the case, and this family's
FINDINGS.md is left untouched -- it records what the tester found at the time, and
that record was accurate when written.
To be clear about what was NOT wrong: A8 does not fail in normal runs and did not
need replacing behaviourally. It passes. The INCONCLUSIVE output it produces under
this sweep is the sweep deliberately cutting its 45s arm wait to 45ms to prove the
honesty machinery fires -- the detector working, not the case breaking.
WINDOWS BEHAVIOUR OF THE SWEEP
On Windows tests/watchdog/grok reports INERT rather than failing: it is wired, but
its starvation-sensitive cases are platform-skipped there, so there is nothing for
starvation to break. Reporting that as a failure would be the gate crying wolf; the
static wiring check is what separates it from a genuine WIRING GAP. Verified on
Windows: 3/3, one INERT, two covered.
pjpoulose
force-pushed
the
fix/watchdog-deadman-resume
branch
from
July 26, 2026 23:04
ec4b004 to
483306e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.