Feature Runtime Verification

Feature: Runtime Verification (opt-in, sandboxed)

Turns needs_runtime_verification findings (and re-tests confirmed ones) into HTTP-level confirmed/refuted with a real PoC — the class of evidence that separated a reference manual audit (5 live confirmations against a real target) from a pure-static tool. It catches guards that exist in code but aren't actually wired at runtime, and it raises precision: a finding that won't reproduce live is honestly downgraded.

Status: opt-in (--runtime, default off) and best-effort — it gracefully skips when it cannot provision a runnable instance. It never blocks a run and never weakens the static pipeline.

The safety crux — why this doesn't break the core guardrail

Argo's hardest guardrail is "no stage ever contacts, scans, or exercises a live in-scope host." Runtime verification preserves it by construction: it never touches the program's real hosts. It builds the open-source target from the already-cloned source into an ephemeral, egress-blocked (--network=none), local instance, and probes only that loopback instance — the same trust model a researcher uses when they build the OSS app and test their own copy.

Risk	Structural prevention
Sandbox reaches the internet / the program's prod hosts	container runs egress-blocked; probes run inside against 127.0.0.1
A probe aimed at a real host (typo/model error)	assert_loopback_only — every target must be loopback; abort otherwise
DoS vs. the local instance	request-count/RPS/payload-size caps; fuzz/flood shapes rejected
Model runs arbitrary shell	the model never gets a shell or network — it emits a constrained probe plan; a fixed executor runs it
Malicious target source executes locally	sealed container, non-root, --rm, no host mounts beyond the source copy + result file
Persistent side effects	throwaway container + ephemeral in-container DB — nothing survives

The LLM's constrained role: propose → validate → execute → interpret

1. Propose — per finding, an HTTP probe plan against localhost:PORT + an expected observation. No shell, no non-loopback hosts.
2. Validate (deterministic) — loopback-only, safe method set, payload/rate/count/timeout caps. Reject → the finding just keeps its static verdict.
3. Execute (fixed in-container runner, not model-controlled) — runs the validated requests.
4. Interpret (model, read-only) — returns confirmed/refuted/inconclusive + evidence, merged into the finding's verdict.

Provisioning — the real blocker, not safety

Getting a complex app to actually run is the hard part. In priority order: an explicit --runtime-image/--runtime-run-cmd, or a committed argo-runtime.json recipe at the repo root ({image | dockerfile, run_cmd?, port?, env?, mount_source?, boot_timeout?}), or the repo's own Dockerfile (auto-built). None available → the stage no-ops; findings keep their static verdict.

Authenticated probes (R2-auth)

A finding entry may carry an optional auth login step (username/password POST), used when credentials are provided — the probe runner keeps a per-finding cookie jar so authz findings can be runtime-confirmed as that session. Known limitation: this does not cover an OAuth authorization-code + PKCE flow (code_verifier/code_challenge are generated at runtime and can't be scripted statically) — deliberately not built, since those findings are already confirmed statically and diffed against a baseline-correct reference; they stay runtime_inconclusive honestly rather than falsely confirmed.

In the report

Each finding's runtime verdict + evidence renders in REPORT.md (confirmed ones also in the submission draft); a gated exec-summary line counts runtime-confirmed findings.

Related

• Adversarial Validation — the static verdict this stage upgrades or confirms.
• Live Verification — the live analog of this exact propose→validate→execute→interpret shape, but against the program's real host instead of loopback.
• Remediation & Fixes — reuses the same isolated-copy build machinery.