feat(core): CLI QoS hardening — drain, lockfile, structured errors, redaction (PER-7855)

[Shivanshu-07]

Summary

Consolidated PER-7855 — proactive CLI hardening (no incident driving it; YAGNI applies). Three logically separable units packaged as one PR with three commits so the diff stays reviewable while the change history reflects the original phased risk-sequencing.

Commit	Topic	Touches
1️⃣ 36bf4b4e	network refactors + redaction + idle-timeout hint (R4/R5/R6/R7)	core/src/{network,utils}.js
2️⃣ 590f845d	per-port lockfile with stale-lock reclaim (R2)	core/src/{lock,percy}.js (new file)
3️⃣ f3261353	SIGINT/SIGTERM drain + unhandled-rejection redaction + bonus child-tree-kill fix (R1/R3)	cli-command/, cli-exec/, cli-snapshot/, core/src/{server,browser}.js

Commit 1 — network refactors

• R4 Move Network.TIMEOUT from a static class field to a per-instance networkIdleWaitTimeout. Concurrent pages with different env values no longer overwrite each other.
• R5 Export AbortCodes enum (ABORTED, TIMEOUT_NETWORK_IDLE). Throws from Network#send for aborted requests now carry {code, reason} via the existing AbortError class. The consumer at network.js:529 prefers error.code === 'ABORTED'; legacy string-match clauses retained for BC.
• R6 Wrap redactSecrets() around the warn/debug logs in executeDomainValidation so upstream errors that echo response bodies don't leak AWS keys, URL-embedded credentials, etc.
• R7 Append actionable hint to network-idle timeout: Hint: set PERCY_NETWORK_IDLE_WAIT_TIMEOUT to increase the budget, or allowlist slow domains via the discovery config.

Implementation note — the _throwTimeoutError path uses a plain Error with code/reason (not AbortError), because error.name === 'AbortError' is checked at discovery.js:520, percy.js:347, and snapshot.js:472 and would silently swallow the timeout as if it were a deliberate cancel. Only the explicit browser-cancellation path uses AbortError.

Commit 2 — per-port lockfile

• R2 New core/src/lock.js: acquireLock({port}) writes ~/.percy/agent-<port>.lock atomically via wx. Payload {pid, port, startedAt}; mode 0o600 on the file, 0o700 on the parent dir.
• LockHeldError carries {meta, lockPath} so the refusal message can name the live pid + lock path for manual cleanup.
• Stale-lock reclaim via process.kill(pid, 0) liveness probe: ESRCH = dead → reclaim; EPERM = alive-but-foreign → refuse; self-pid → reclaim (we cannot conflict with ourselves).
• Reclaim is unlink + retry-wx, not rename-based: Windows CI is pinned to Node 14 (.github/workflows/windows.yml:15) where fs.renameSync over an existing target is unreliable.
• Percy.start() acquires the lock as the first step inside try {, before any expensive setup; registers process.on('exit') synchronous unlink as last-chance cleanup.
• Percy.stop() releases the lock in the finally block (idempotent).
• Backwards compatibility: when the lock is held, the catch maps LockHeldError to the legacy Percy is already running or the port X is in use message string (downstream tooling may grep for it) AND also log.errors the actionable detail.

Commit 3 — graceful drain + unhandled-rejection redaction

• R1 New module-level shutdownState bag exposed to commands as ctx.shutdown so they can call percy.stop(ctx.shutdown.forced) for graceful-on-first-signal, force-on-second-signal behavior.
• First SIGINT/SIGTERM: log ${signal} received, draining (press Ctrl-C again to force)..., arm 30s drain timer.
• Second signal (or 30s timer): flip forced=true, arm 5s hard-exit safety timer.
• Production exit codes: SIGINT→130, SIGTERM→143 via process.exit only when definition.exitOnError is true; tests with exitOnError: false preserve the legacy clean-resolution.
• R3 Global unhandledRejection / uncaughtException handlers, attached exactly once. Stack trace routed through redactSecrets() so CDP rejections that include serialized page-script bodies, Authorization headers, or cookie strings cannot leak. activeContext.runFailed=true ensures non-zero exit even when the rejection is non-fatal.
• Bonus Fixed the existing POSIX child-tree leak in core/src/browser.js:207. The previous this.process.kill('SIGKILL') targeted only the lead Chromium pid despite detached: true at :266, leaving renderer/utility/zygote children orphaned on every kill. Fix matches Puppeteer / Playwright convention: taskkill /pid <pid> /T /F on Windows, process.kill(-pid, 'SIGKILL') on POSIX (negative-pid signals the process group). Falls back to lead-pid kill on either path's error.
• HTTP server graceful drain: Server.close() becomes async with drainMs (default 5s), uses Node 18.2+ closeIdleConnections/closeAllConnections with Node 14 fallback.

Tests

• 23 net-new specs across the three commits:
  • 6 in core/test/unit/{network,utils}.test.js (SC6 per-instance timeout, R5 AbortCodes shape, SC8 redactSecrets fixtures)
  • 13 in core/test/unit/lock.test.js (SC3 stale reclaim, SC4 live-foreign refusal, SC5 multi-port, EPERM-as-alive, corrupt-payload recovery, mkdir-p, mode bits on POSIX, release idempotency, re-acquire after release)
  • 4 in cli-command/test/shutdown.test.js (SIGINT→130, SIGTERM→143, shutdown.forced transition, redactSecrets path for unhandled rejections)
• Updated core/test/discovery.test.js, cli-command/test/command.test.js, cli-exec/test/exec.test.js for the new "draining" announcement on stderr, the removal of the legacy "Stopping percy..." log on graceful interrupts, and the AbortCodes/idle-hint message changes.
• Test infrastructure: added ~/.percy/agent-* to mockfs $bypass (lock files use real fs); _resetShutdownForTest() exported from @percy/cli-command for spec isolation; module-level shutdown state auto-resets at the start of each runCommandWithContext; try/finally in runCommandWithContext ensures per-run signal listeners are always removed (eliminates pre-existing MaxListenersExceededWarning).

Test run on this branch (sequential, per workspace)

Workspace	Specs	Pass	Fail	Notes
@percy/core	703	676	27	Same 27 pre-existing failures as master baseline (21 install Chromium, 5 runDoctorOnFailure, 1 API server when disabled). All 19 new specs pass.
@percy/cli-command	62	62	0	All 4 new shutdown specs pass; 1 pre-existing test ("handles interrupting generator actions") still passes after assertion update.
@percy/cli-exec	33	33	0	2 tests updated for Phase 3 behavior changes (drain announcement, removed force-stop log).

⚠ Important — running tests: @percy/core and @percy/cli-exec both bind port 5338 via Percy.start(). With Phase 2's lockfile, running these two suites in parallel will fail the second-to-acquire with LockHeldError — that is the lockfile working as designed, refusing concurrent same-port starts across processes. Run the workspace test suites sequentially, or set distinct PERCY_SERVER_PORT per worker if you parallelize CI. Same for any developer running lerna run --parallel test.

(Pre-existing in cli-snapshot/test/file.test.js: 4 failures from a mockfs/dynamic-import() resolver issue unrelated to this PR; identical on master.)

Test plan

• CI green on Linux + macOS + Windows modulo the 27 pre-existing failures
• Manual: trigger network-idle timeout, confirm hint appears in stderr
• Manual: rm -rf ~/.percy/, percy start, kill -9, percy start again — second succeeds via stale-lock reclaim
• Manual: percy start in two terminals on the same port — second refuses with the actionable message naming pid + lock path
• Manual: percy start --port 5338 and percy start --port 5339 concurrently — both succeed
• Manual: percy start, Ctrl-C → drain message + clean exit 130
• Manual: percy start, Ctrl-C, Ctrl-C again → forced exit ≤ 2s, no orphan Chromium (ps -A | grep -i chrom empty)
• Manual: kill -TERM <pid> → exit 143
• Windows manual: console Ctrl+C → drain message + clean exit (SIGINT works on Windows; SIGTERM is documented as best-effort because Windows can't deliver it gracefully)

Risks

Risk	Mitigation
Tests that emit process.emit('SIGINT') and expect empty stderr	Updated to expect the drain announcement
Tests that expect Stopping percy... log on signal interrupt	Updated — graceful drain doesn't force-stop, so that log doesn't fire
process.exit(130) in test mode kills the test runner	Production-only via definition.exitOnError gate
Browser child-tree kill change (POSIX negative-pid) might fail in containers	Fallback to lead-pid SIGKILL preserves at-least-as-good-as-before behavior
Drain hangs at Percy.stop(false)	5s hard-exit safety timer after second signal / 30s drain timeout
Lock file leaks on hard kill (SIGKILL)	Reclaimed on next start via process.kill(pid, 0)
Multi-user host: another user's pid happens to match	Treated as alive (EPERM); user must manually delete the file (path is in the refusal message)
Restricted CI without writable $HOME	acquireLock propagates EACCES via the catch path with an actionable message; future ticket can add tmpdir fallback if real users hit this
Concurrent same-port test workers	Phase 2 refuses with LockHeldError by design; document running tests sequentially or per-worker PERCY_SERVER_PORT
secretPatterns.yml doesn't cover Cookie:/JSESSIONID/custom-auth	SC8 acceptance covers stated categories only; yml augmentation is a separate ticket

Post-Deploy Monitoring & Validation

• Logs to watch (24h–1 week post-merge):
  • Orphaned Chromium reports in #percy-cli Slack — should drop to zero (Phase 3 bonus + R1)
  • Any LockHeldError in build logs — expected to drop after legitimate stale locks self-reclaim once
  • Build logs containing unredacted Authorization: / AKIA* / URL credentials — should drop to zero (R6 + R3)
  • Any [REDACTED] markers in unhandled-rejection logs — confirms the new redaction path is live
  • PERCY_NETWORK_IDLE_WAIT_TIMEOUT related support tickets — should decrease as users hit the new hint
• Validation checks:
  • ls ~/.percy/ after a clean percy start && percy stop — should be empty
  • ps -A | grep -i chrom after a SIGINT'd percy start — must be empty (POSIX) / tasklist | findstr chrome empty (Windows)
• Failure signal(s) / rollback trigger:
  • Reports of stuck builds that never exit on Ctrl-C — drain hang
  • Chromium children accumulating after Ctrl-C
  • "Percy is already running" errors when no Percy is running and ~/.percy/agent-X.lock is present (PID-reuse false positive on long-running hosts)
• Validation window & owner: 1 week post-merge; @shivanshu.si

Origin / Plan

• Origin requirements: docs/brainstorms/2026-04-24-per-7855-cli-qos-hardening-requirements.md
• Plan: docs/plans/2026-04-27-001-feat-per-7855-cli-qos-hardening-plan.md

This PR consolidates the previously-staged drafts: #2196 (Phase 1), #2197 (Phase 2), #2198 (Phase 3) — those will be closed in favor of this single PR.

---

🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via Claude Code