Automated verification pipeline: eliminate human code review through layered trust

[rtzll]

Summary

Rascal today orchestrates coding agents (Claude, Codex, Goose) to generate code from specs and ship the results as pull requests. The current flow is linear and has no verification between "agent finished writing code" and "PR created on GitHub":

Spec + Agent → Run → PR (human reviews)

The goal of this issue is to build a multi-layered automated verification pipeline that makes human code review unnecessary for the majority of changes. The conceptual model is the Swiss cheese model — no single verification layer is perfect (each has "holes"), but stacking multiple independent layers ensures that what slips through one is caught by the next.

The target flow becomes:

Spec + Agents → Compare Options → Deterministic Guardrails → Acceptance Criteria → Permission Systems → Adversarial Verification → PR (auto-merge)

Each layer is described in detail below with context on how it fits into rascal's existing architecture.

---

Background: How Rascal Works Today

For an implementer unfamiliar with the codebase, here's the relevant architecture:

• rascal (CLI): User-facing tool for creating runs, viewing logs, managing config.
• rascald (orchestrator): HTTP API server that receives requests, persists state to SQLite, schedules runs, and supervises execution in Docker containers.
• rascal-runner (worker): Runs inside a Docker container. Clones the repo, invokes the configured agent runtime, collects artifacts (agent.ndjson, commit_message.txt, pr_body.md), commits, pushes a branch, and creates/updates a PR.

Key internal packages:

• internal/orchestrator/ — scheduling, supervision, API endpoints, webhook handling
• internal/worker/worker.go — in-container execution logic (clone → agent → commit → push → PR)
• internal/runner/ — Spec struct defining the input contract for a run (RunID, TaskID, Repo, Instruction, AgentRuntime, etc.), launcher interface (Docker/Noop)
• internal/agent/ — Runtime (goose-codex, codex, claude, goose-claude), Harness (goose vs direct), ModelProvider (codex vs anthropic)
• internal/state/ — SQLite persistence for runs, tasks, credentials, sessions

A run today goes: queued → running → succeeded/failed/review. There are no quality gates between "agent completed" and "run finalized."

---

Layer 1: Compare Multiple Options

What it is

Instead of running a single agent on a task and hoping for the best, run N agents (or the same agent N times) on the same spec in parallel and compare the results. Pick the best solution or synthesize a consensus.

Why it's needed

LLM outputs are non-deterministic. A single generation might take a suboptimal approach, miss an edge case, or hallucinate an API. Running multiple generations and comparing creates a natural selection pressure — good solutions tend to converge, and outlier failures are easy to spot.

How it fits into rascal

• Rascal already supports multiple runtimes (claude, codex, goose-codex, goose-claude). The infrastructure to run different agents on the same repo exists.
• The runner.Spec already carries everything needed (instruction, repo, branches). Fan-out means creating N specs from one task.
• The orchestrator's per-task serialization (internal/orchestrator/) would need to be relaxed to allow parallel runs within a comparison group.

Implementation sketch

• Add a Strategy field to the task or run config: single (current default) vs compare(n=3, runtimes=[...]).
• In internal/orchestrator/, when strategy is compare, create N runs and wait for all to complete.
• Add a Comparator phase that receives N diffs and selects/merges. This could be:
  • Deterministic: pick the diff that passes the most downstream layers (guardrails, criteria).
  • LLM-judged: feed all diffs to a judge agent that picks the best.
  • Consensus: if N-1 solutions agree on a change, prefer that over the outlier.
• Only the selected solution proceeds through the remaining layers.

Trade-offs

• Cost: Nx compute per task. Best reserved for high-stakes changes or when cheaper layers keep rejecting single attempts.
• Complexity: comparing diffs is non-trivial for large changes.

---

Layer 2: Deterministic Guardrails

What it is

Hard, deterministic checks that run after the agent completes but before a PR is created. These are fast, cheap, and merciless — binary pass/fail with zero tolerance.

Why it's needed

Agents regularly produce code that doesn't compile, fails tests, introduces lint violations, or accidentally includes secrets. These failures are trivially detectable by machines and should never reach a human reviewer. Deterministic guardrails are the highest-ROI layer because they catch the most common and most obvious failures at near-zero cost.

How it fits into rascal

• The rascal-runner container already has the full repo checked out and the agent's changes committed locally. Running go test, npm test, a linter, or a secret scanner at this point is straightforward.
• The worker (internal/worker/worker.go) currently goes: agent execution → read artifacts → finalize. A guardrails phase slots in between agent execution and finalization.
• If any guardrail fails, the run is marked failed with a structured reason (which guardrail, what output), and no PR is created.

Implementation sketch

• Add a Guardrails phase in internal/worker/worker.go after the agent subprocess completes.
• Guardrails are configured per-repo via .rascal/guardrails.yaml:

  guardrails:
    - name: tests
      command: "go test ./..."
      timeout: 300s
    - name: lint
      command: "golangci-lint run"
      timeout: 60s
    - name: secrets
      command: "gitleaks detect --source . --no-git"
      timeout: 30s
    - name: build
      command: "go build ./..."
      timeout: 120s
    - name: banned-patterns
      patterns:
        - "os\\.Exit" # don't call os.Exit outside main
        - "fmt\\.Print" # use structured logging
      scope: diff-only

• Each guardrail runs in sequence (or parallel where safe). On failure, the run record gets a structured guardrail_results field with name, status, and output.
• The runner already writes meta.json with run results — extend this to include guardrail outcomes.

Checks to consider

• Build: does the code compile?
• Tests: do existing tests pass? Do new tests exist for new code?
• Lint/format: does the code meet style standards?
• Secret scanning: no API keys, passwords, or tokens in the diff (trufflehog, gitleaks).
• Banned patterns: regex-based rules on the diff (e.g., no fmt.Println in library code).
• Diff size limits: reject changes over N lines as likely scope creep.

---

Layer 3: Acceptance Criteria

What it is

Structured, task-specific criteria that the agent's output must satisfy. Unlike guardrails (which are repo-wide and generic), acceptance criteria are defined per-task and describe what "done" looks like for that specific change.

Why it's needed

A change can pass all guardrails (builds, tests pass, no lint issues) and still be wrong — it might implement the wrong behavior, miss a requirement, or solve a different problem than what was asked. Acceptance criteria bridge the gap between "code is valid" and "code is correct."

This is the layer that encodes the reviewer's question: "Does this actually do what was requested?"

How it fits into rascal

• Currently, runner.Spec has an Instruction field — a plain string prompt. This is the only input the agent gets. There's no structured way to express what success looks like.
• The spec format needs to evolve from a plain instruction to a structured document with verifiable conditions.

Implementation sketch

• Extend runner.Spec (or add a new field alongside Instruction):

  type Spec struct {
      // ... existing fields ...
      Instruction        string
      AcceptanceCriteria []Criterion
      Constraints        []Constraint
  }
  
  type Criterion struct {
      Description string // "The /health endpoint returns 200 with {status: ok}"
      Type        string // "test", "grep", "llm-judge", "file-exists"
      Check       string // type-specific: test name, grep pattern, judge prompt, file path
  }
  
  type Constraint struct {
      Description string // "Do not modify the database schema"
      Type        string // "no-touch-files", "no-new-deps", "max-lines"
      Value       string // glob pattern, number, etc.
  }

• A CriteriaChecker runs after guardrails, evaluating each criterion:
  • test: run a specific test and check it passes.
  • grep: check the diff or codebase for expected patterns.
  • file-exists: verify expected files were created.
  • llm-judge: feed the diff + criterion to an LLM and ask "does this change satisfy: {criterion}?" — more expensive but handles nuanced requirements.
• Results are stored per-criterion on the run record and rendered as a checklist in the GitHub comment:

  ## Acceptance Criteria
  - [x] The /health endpoint returns 200
  - [x] Added unit test for health handler
  - [ ] Updated OpenAPI spec ← FAILED: no changes to openapi.yaml
  

• If any required criterion fails → run status = failed, no PR.

Where criteria come from

• GitHub issue body (parsed from a structured section).
• .rascal/specs/ directory with YAML/Markdown spec files.
• Inline in the instruction with structured markers.
• The agent itself could be asked to propose criteria before starting (two-phase: plan → execute).

---

Layer 4: Permission Systems

What it is

A scoping mechanism that restricts what the agent is allowed to do, independent of what it was asked to do. Even if the instruction says "refactor the auth module," permissions can enforce that only internal/auth/ files are touched.

Why it's needed

Agents are susceptible to prompt injection (especially when processing issue bodies or PR comments from untrusted users), scope creep (making "helpful" changes beyond the request), and accidental damage (deleting files, modifying CI config, changing dependencies). Permissions create a blast-radius limit.

This is defense-in-depth: even if an agent is tricked into doing something harmful, the permission system prevents the harm from landing in the PR.

How it fits into rascal

• The runner container has full control over the workspace. The diff is created inside the container before being pushed. This is the ideal enforcement point — validate the diff against permissions before git push.
• Rascal already has a permissions/auth concept for users and credentials (internal/credentials/), but nothing scoping what the agent can do to the repo.

Implementation sketch

• Per-repo or per-task permission config (.rascal/permissions.yaml):

  permissions:
    allowed_paths:
      - "internal/auth/**"
      - "internal/auth_test.go"
    denied_paths:
      - ".github/**"
      - "go.mod"
      - "go.sum"
      - "*.sql" # migrations need human review
    max_files_changed: 10
    max_lines_changed: 500
    allowed_operations:
      - modify
      - create
    denied_operations:
      - delete
    forbidden_imports:
      - "unsafe"
      - "os/exec"
    forbidden_commands: # what the agent can run in the container
      - "curl"
      - "wget"

• Enforcement in internal/worker/worker.go after agent execution:
  1. Parse the git diff.
  2. Check every changed file against allowed_paths / denied_paths.
  3. Check operations (create/modify/delete) against allowed_operations.
  4. Check diff stats against max_files_changed / max_lines_changed.
  5. Scan new imports against forbidden_imports.
• Violations result in run failure with specific details (which file, which rule).
• Permissions can also be derived dynamically from the task context — e.g., if the trigger is a PR review comment, scope permissions to only the files in that PR.

Levels of strictness

• Strict mode: explicit allowlist, everything else denied. For high-trust automated pipelines.
• Advisory mode: log violations but still create PR with warnings. For gradual adoption.
• Dynamic scoping: derive permissions from the trigger context (e.g., issue labels, PR file list).

---

Layer 5: Adversarial Verification

What it is

A separate, independent agent whose sole job is to find problems with the primary agent's output. It acts as an automated adversarial reviewer — rewarded for finding bugs, security issues, and spec violations, not for shipping code.

Why it's needed

This is the most powerful layer because it catches the subtle issues that deterministic checks miss: logic errors, race conditions, security vulnerabilities, incorrect business logic, missing error handling, and spec misinterpretations. It simulates the value a human reviewer provides, but automated and consistent.

The key insight is that reviewing code is easier than writing code — a reviewer agent with the spec and diff can focus entirely on critique without the cognitive load of implementation.

How it fits into rascal

• Rascal already has the infrastructure to run agents in containers with repo access. The reviewer is essentially another run, but with a different instruction (review prompt) and different input (the diff rather than the original spec).
• The orchestrator can chain runs: primary run → if passed layers 2-4 → verification run → if approved → finalize PR.
• Using a different model or provider than the primary agent adds diversity (a Claude-generated change reviewed by Codex, or vice versa), reducing correlated failures.

Implementation sketch

• Add a Verification phase in the orchestrator, triggered after a run passes guardrails, criteria, and permissions.
• The verification run gets a structured prompt:

  You are a code reviewer. Your job is to find problems.
  
  ## Original Spec
  {instruction + acceptance_criteria}
  
  ## Diff
  {git diff}
  
  ## Your Task
  Review this change for:
  1. Correctness: Does it do what the spec asks?
  2. Bugs: Logic errors, edge cases, off-by-ones, nil dereferences.
  3. Security: Injection, auth bypass, data leaks, SSRF.
  4. Performance: N+1 queries, unbounded loops, memory leaks.
  5. Compatibility: Breaking changes, API contract violations.
  
  Output a JSON verdict:
  {
    "verdict": "approve" | "request_changes",
    "issues": [{"severity": "critical|major|minor", "location": "file:line", "description": "..."}]
  }
  

• If verdict is request_changes:
  • Auto-retry mode: feed the issues back to the original agent as a follow-up instruction, re-run, and re-verify (with a max retry count to prevent loops).
  • Fail mode: mark the run as failed with the reviewer's issues attached.
• Store verification results on the run record. Render in GitHub comment.

Preventing collusion / correlated failures

• Use a different model provider for verification than for generation.
• The reviewer prompt should be adversarial — explicitly incentivize finding problems.
• Consider running multiple reviewer agents and requiring consensus.

---

Implementation Priority

Recommended build order based on impact-to-effort ratio:

Priority	Layer	Effort	Impact	Why this order
1	Deterministic Guardrails	Low	High	Catches the most failures at lowest cost. Foundation for everything else.
2	Permission Systems	Low	High	Cheap to enforce, prevents catastrophic scope issues and prompt injection damage.
3	Acceptance Criteria	Medium	High	Requires spec format evolution, but makes the system actually verify correctness.
4	Adversarial Verification	Medium	Very High	Most powerful layer; catches subtle issues. Requires infra for chained runs.
5	Compare Multiple Options	High	Medium	Most expensive (Nx compute). Best value once other layers exist to score candidates.

Each layer should be independently toggleable per-repo via .rascal/ config files and buildable as a standalone feature that adds value even without the other layers.

---

Architecture Considerations

Config structure

All layer configs should live in .rascal/ in the target repo:

.rascal/
  guardrails.yaml      # Layer 2
  permissions.yaml     # Layer 4
  verification.yaml    # Layer 5 config (which model, retry count, etc.)
  specs/               # Layer 3 spec templates

Data model changes

• runs table needs: guardrail_results JSON, criteria_results JSON, permission_violations JSON, verification_verdict JSON.
• New run_phases table or embed phase tracking in meta.json.
• tasks table may need acceptance_criteria and strategy fields.

Execution flow (target state)

1. Task created (instruction + criteria + permissions + strategy)
2. If strategy=compare: fan out N runs, else: single run
3. Agent executes in container
4. [Layer 2] Guardrails run in container → fail fast if violated
5. [Layer 4] Permission check in container → fail fast if violated
6. [Layer 3] Acceptance criteria checked (in container or orchestrator)
7. If strategy=compare: comparator selects best passing solution
8. [Layer 5] Adversarial verification run (separate container)
9. If approved → push branch, create/update PR, mark succeeded
10. If rejected → optionally retry with feedback, or mark failed

Observability

Each layer should produce structured output that's:

• Stored on the run record for API/CLI access.
• Rendered in the GitHub PR comment so stakeholders see what was checked.
• Available in runner logs for debugging.

---

Success Criteria for This Issue

• At least layers 1-2 (guardrails + permissions) are implemented and configurable per-repo
• The worker executes verification phases between agent completion and PR creation
• Run records include structured results from each layer
• GitHub PR comments show which layers passed/failed
• A repo can opt into the full pipeline via .rascal/ config files
• Documentation covers how to configure each layer