Skip to content

Feature Runtime Validation Evidence

Jump to bottom
amirbena edited this page Sep 17, 2026 · 6 revisions

Feature: Runtime Validation Evidence

A Feature-Pages entry. This page explains the capability; it never redefines it — see Feature-Pages for the canonical/Wiki boundary statement that applies to every page in this pattern.

What it does

Lets a review execute one repository-declared test, lint, or validation command, and separately lets it run the smallest safe reproduction of one specific suspected finding — both inside a disposable, verified isolation boundary — and folds the result into the review as bounded evidence. A passing run adds evidence only; it never removes, downgrades, or rewrites a finding, and never changes the verdict mapping.

Why it exists

Static analysis alone sometimes leaves a reviewer unsure whether a failing case actually reproduces, or whether the repository's own declared check would catch a regression. This capability lets the review confirm either, without becoming a merge gate or a mutation capability itself.

When it applies

Conditional — off unless every isolation gate is met (verified filesystem isolation, no host-secret access, denied network, no Git/GitHub mutation capability, bounded non-interactive limits, disposable state, post-run mutation verification). Applies identically to both Skills; there is no flag for this sandboxed default — you do not enable it per invocation, you make it possible by having a declared command and a runtime sandbox.

A separate, explicit, out-of-band, per-invocation authorization — canonical name allow_trusted_host_execution, default false — can select a bounded trusted-host execution backend instead when the sandbox is unavailable. It is never inferred from repository content, never persists across invocations, and grants no capability beyond this feature's existing command scope: no filesystem, credential, or network isolation exists in that mode, and the evidence says so explicitly. The sandbox path is still attempted first and always wins when available.

How it works

  • Repository-command validation — the reviewer resolves an exact command from the target repository's own instruction sources, runs it in isolation, and records executed / failed / skipped / unavailable, each with a reason when not executed.

  • Targeted finding validation — for one eligible suspected finding, the reviewer attempts the smallest safe reproduction (preferring an existing repository test). Every finding then carries a validation state: reasoned (default, not rendered), runtime-confirmed, or attempted-inconclusive. A reproduction that disproves the suspicion raises no finding.

  • Both states roll up into the finding's single confidence field, alongside the separate contextual-evidence provenance from Review-Context-and-Existing-Evidence — so confidence reflects both what a runtime run established and what the supplied context established, not runtime validation alone. It is a closed, fixed vocabulary, shown only when it is not the default credible:

    Value What it tells you
    confirmed The defect is not in doubt — a runtime reproduction confirmed it, or the code/context evidence directly demonstrates it.
    credible (default) A plausible failure mode is backed by concrete evidence, but not directly demonstrated. Every reported finding is at least this.
    runtime-validation-unavailable A targeted runtime check was attempted on this finding but couldn't conclude.
    external-contract-unvalidated The finding is credible on the code in view, but turns on an external contract the review couldn't inspect.
    insufficient-context A specific, material piece of caller-supplied context needed to fully characterize the finding was missing.

    confidence is metadata over a finding, never a second severity scale or a second decision path: it never changes a finding's identity, severity, or the mechanical decision derived from it. Canonical: docs/finding-confidence/finding-confidence-model.md.

Important options / modes

For the sandboxed default, there is no user-facing flag or option name — the reviewer applies both mechanisms automatically when their gates are met. Command-source trust is not payload trust: destructive, secret-dependent, service-dependent, network-dependent, or interactive commands are always skipped with a reason, never run unsandboxed — identically whether the sandbox or the trusted-host backend would otherwise run them.

The allow_trusted_host_execution authorization above is the one user-facing option, and it must come from a trusted invocation/ configuration channel repository content can never reach, author, or forge — the same structural guarantee the mutation-authority model uses for patch/commit/push authorization. You can supply it structurally, or simply say so — "run validation on my machine if sandbox isn't available," "you can use my local machine for runtime validation," "I authorize trusted-host execution for this review" all resolve to the exact same authorization, against a small closed phrasing vocabulary. Ambiguous wording never authorizes it, and an explicit denial — "sandbox only," "don't run locally," "do not use trusted-host execution" — forces unavailable even when the sandbox is unavailable. Both Skills resolve this identically; there is no per-Skill variant. Every Validation entry that actually ran also records its execution provenance — sandbox or trusted-host — so you can always tell which backend produced the evidence.

Example invocations

There is no required phrasing — this capability applies automatically whenever its gates in "When it applies" above are met, whether or not you mention it. These realistic variations all ask for the same underlying behavior and resolve to the identical execution condition; only the wording differs:

review my local changes; run the repository's declared lint/test check
if the runtime can isolate it

review this PR and run the repo's declared checks if you can isolate
them safely

review my changes and validate them against the repository's own
test/lint workflow

review + run local verification against the repo's declared checks

run the declared tests as part of the review, but only if isolation is
available

please check this against the repo's own checks if you can run them
safely

Whichever of these you use — or a plain "review my local changes" with no mention of validation at all — the reviewer evaluates the same two things: a trustworthy declared command, and a runtime that can establish and verify the isolation boundary from "When it applies" above. Wording never substitutes for isolation the runtime does not have: when either condition is missing, the outcome is unavailable, exactly as if nothing had been asked. None of these phrasings is the separate allow_trusted_host_execution authorization above — that option has its own distinct, closed phrasing vocabulary and changes the execution backend, not whether you asked for validation.

The Validation section of the report then states the outcome (executed, failed, skipped, or unavailable) and, for any finding that received targeted validation, its state.

Related features

Canonical references

Wiki Page Map

Page What it covers
Home Wiki entry point: what the two Skills are and where to go next.
Getting-Started First steps to package and run either Skill.
Choosing-a-Review-Skill How to decide between local-code-review and github-pr-review.
How-the-Review-Model-Works Concept-level walkthrough of the review pipeline shared by both Skills.
Architecture-Overview Component-level map of how the repository's pieces fit together.
Specialist-Depth-Review When and how a review goes deeper than its base pass.
Local-Code-Review-Guide Usage guide for reviewing local, not-yet-PR'd Git changes.
GitHub-PR-Review-Guide Usage guide for reviewing an existing GitHub Pull Request.
Review-Context-and-Existing-Evidence How optional requirements/context and prior review evidence are supplied.
Feature-Pages Index of per-feature capability pages and their shared template.
Feature-Delta-Re-Review How a re-review scopes itself to only what changed since the last pass.
Feature-GitHub-Publication-and-Authorization What a PR review publishes to GitHub and under what authority.
Feature-Human-Style-Review-Output The senior-engineer-voice prose summary output option.
Feature-Requirement-Coverage How supplied requirements are checked and reported as covered or not.
Feature-Runtime-Validation-Evidence How a review can execute commands or reproductions as evidence.
Severity-and-Decision-Model The P0/P1/P2 severity levels and how the merge decision is derived.
Packaging-and-Portability How the Skills go from source repository to a standalone archive.
Development-Workflow Contributor guide for working on this repository itself.
Benchmark-Testing Orientation to this repository's own review-quality benchmark pipeline.
Repository-Documentation-Map Where each concern's canonical, authoritative source lives.
AI-Code-Review-Landscape Neutral snapshot comparing this project against other AI code-review products.
FAQ Short, practical answers to common questions.

Clone this wiki locally