Forge

A longitudinal control plane for autonomous software engineering.

Forge is not a coding agent. It sits above coding agents — Claude Code, Codex, Pi, whatever comes next — and treats them as interchangeable engineering workers. Its job is to assign work, isolate it, measure the result independently, and remember what happened, so that engineering decisions can eventually be made from evidence instead of vibes.

The question most systems answer is did the task pass? The question Forge is built to answer is:

Did this change actually make the software better — and which agent, configuration, and context is most likely to make it better next time?

See forge_project_plan.md for the full design.

---

Status

Forge can run a real engineering task through Claude Code or Codex in isolated worktrees, evaluate each result itself, and group independent attempts from one base commit into a comparative experiment.

Capability	State
Core domain model (task, run, event, workspace, evaluation, metric)	✅
Git worktree isolation with safety invariants	✅
Subprocess execution: timeouts, output caps, process-group cleanup	✅
Environment policy and secret redaction	✅
Typed test, benchmark, lint, security, complexity, and custom evaluation	✅
SQLite experience ledger (runs, trajectories, evaluator results, metrics)	✅
forge init, forge agent list, forge task validate	✅
Claude Code adapter and forge run	✅
Protected evaluation inputs, candidate patch policy, security posture	✅
Structured benchmark metric contract	✅
Codex adapter and forge run --agent codex	✅
forge compete task.yaml --agents claude,codex	✅
History, agent statistics, failures, similarity, experiments, JSONL export	✅
Provider-agnostic historical routing and trusted evidence policy	✅
Task-driven multi-agent DAG execution with final evaluation	✅
Immutable, commit-bound Repository World Model	✅
Longitudinal repository health and typed trends	✅
Immutable, evidence-backed engineering policy optimization	✅
Empirical Tier 1 campaign (20 real paired tasks)	✅ completed; router selected 0 at the frozen 0.05 margin
Production hardening (OCI containment, recovery, CI/release tooling)	✅ 1.1.0 local release candidate; 7/7 executed RC4 tasks PASS, 2 human-waived; GitHub CI/publication pending

---

Quickstart

Prerequisites

• Rust 1.93 or newer (cargo build --release)
• Git 2.5 or newer (worktree support)
• Claude Code or Codex CLI on your PATH, already authenticated. Check exact availability with forge agent list.
• A Git repository with at least one commit. Agents work from a commit, so uncommitted changes are invisible to them.

Run a task

forge init

forge run .forge/tasks/my-task.yaml --agent claude

or, through the identical Forge pipeline and prompt contract:

forge run .forge/tasks/my-task.yaml --agent codex

forge agent list shows which agents Forge can actually run, and forge task validate <file> checks a task before you spend a run on it.

The retained evidence is queryable without writing SQL:

forge history --agent codex --outcome pass --limit 20
forge agent stats codex
forge failures --component storage --agent codex
forge task similar T-1042
forge experiments list
forge export --format jsonl > forge-runs.jsonl

These commands describe recorded evidence. Forge can apply the deterministic historical-baseline-v1 policy without executing an agent:

forge run task.yaml --agent recommend

It selects only when readiness and the score margin permit; otherwise it abstains. Explicit --agent auto is the only routing mode that authorizes execution, and it is not approved for unattended production until the prospective holdout is complete.

Before a supervised production window, run the read-only diagnostic and take a verified store backup:

forge doctor
forge store backup --output /backups/forge.db

Forge also records the engineering policy governing each new run and can build bounded, store-backed proposals without changing evaluation truth:

forge policy show
forge policy propose --max-world-facts 8
forge policy compare PP-0001
forge policy experiment create PP-0001
forge policy promote PP-0002 --actor operator
forge policy rollback P-0001 --reason "observed regression"

Promotion and rollback are explicit. See docs/policy-optimization.md for the guardrail, cutoff, experiment, approval, and reproducibility contract.

To run independent attempts from one resolved base and compare their evidence without choosing an overall winner:

forge compete .forge/tasks/my-task.yaml --agents claude,codex

Competition is sequential in Phase 1B. Each participant is an ordinary Forge run with its own worktree, patch, evaluation, and trajectory. The experiment stores their shared base and run links, then reports pairwise dimensional relationships. One failed or timed-out participant does not erase the others.

To coordinate dependent work from an explicit, reproducible task DAG:

forge team .forge/tasks/my-team-task.yaml

Agent-backed nodes remain ordinary Forge runs in isolated worktrees. Declared artifacts and candidate commits are the only handoff channels, review evidence is advisory, and the integrated candidate is independently checked through the normal patch, integrity, and evaluation pipeline. See docs/team-execution.md for the plan format and execution contract.

To build and query static repository evidence without a live model:

forge world build
forge world show
forge world query component storage
forge world query dependencies storage

World builds require a clean checkout so uncommitted content can never be mislabelled as truth about HEAD. See docs/world-model.md for snapshot, provenance, staleness, extraction, and integration semantics.

Exit codes

Code	Meaning
0	A change was produced and every check passed
1	Forge could not run the task (bad task, missing CLI, not a repo)
2	The run completed but the outcome was not a pass
3	Automatic single-agent routing stopped without selecting an agent

1 and 2 are deliberately different: "the tool broke" and "the change didn't work" call for different responses in a script.

---

A task

A task separates what the agent is told from what Forge measures. The agent never gets a say in the second part.

task_id: T-1042
repository: distributed-runtime
objective: >-
  Improve checkpoint write throughput without weakening recovery guarantees.

constraints:
  - All existing tests must pass
  - Recovery semantics cannot change
  - Memory increase must remain below 10%

evaluation:
  tests:
    command: cargo test --workspace
    timeout_secs: 900
  lint:
    command: cargo clippy --workspace -- -D warnings
  benchmark:
    command: ./bench/checkpoint.sh
    metrics_file: .forge-metrics.json

protected_paths:
  - tests/**
  - benches/**
# A task that legitimately updates a particular test may grant a narrow exception:
allowed_protected_paths:
  - tests/checkpoint_format.rs

classification:
  category: performance
  language: rust
  domain: storage
  difficulty: hard
components:
  - checkpointing
  - storage
tags:
  - throughput

repository must match the name in .forge/config.toml, so a task cannot be run against the wrong repository by accident.

Classification, components, and tags are optional repository-defined strings used for cohorts and deterministic similarity. Forge never asks an LLM to infer them. Existing metadata.task_type, metadata.language, and metadata.subsystem fields remain compatible as category, language, and domain fallbacks.

Each run is bound to an immutable content-addressed snapshot of the complete task definition. Reusing a task ID with edited instructions or classification affects future runs only; historical queries and exports retain the revision that each older run actually executed.

See docs/evaluation.md for every evaluator category, required versus optional policy, structured metric output, and trust rules. See docs/experience-ledger.md for query semantics, similarity weights, missing-data rules, and the JSONL schema. See docs/routing.md for pre-run features, provenance, evidence eligibility, readiness, decision contracts, and reproducibility. See docs/team-execution.md for task DAGs, typed handoffs, assignment, integration, review, and team persistence. See docs/world-model.md for commit-bound architecture facts, deterministic extraction, typed queries, and exact agent context. See docs/phase-7-longitudinal-health.md for missing-data, comparability, attribution, and trend semantics. See docs/policy-optimization.md for immutable engineering policies, evidence-backed proposals, canaries, and rollback. See docs/validation.md for the empirical validation program: the 20-task corpus, the pre-registered analysis rules, and what has actually been measured rather than built. See docs/codex-accounting.md for provider token evidence, versioned Codex credit derivation, missingness, and the offline campaign-enrichment boundary. See docs/routing-validation.md for the exact Rust replay, Tier 1 zero-selection finding, and prospective holdout. See docs/security.md and docs/operations.md for required OCI containment, resource controls, backups, recovery, and incident response. See docs/production-readiness.md for the conservative readiness matrix and docs/external-pilot.md for the frozen supervised-pilot protocol and its RC4 outcome.

---

What a run does

validate task → resolve base commit → create run record
      ↓
create isolated Git worktree from that commit
      ↓
invoke the selected coding agent with the task contract    ← untrusted
      ↓
──────────────────── TRUST BOUNDARY ────────────────────
      ↓
read the full workspace delta relative to the base commit
      ↓
apply candidate patch policy and check protected evaluation inputs
      ↓
run the task's own checks against the workspace
      ↓
derive the outcome → persist run, events, patch, metrics

The report keeps the two judgments apart, because they are different claims from different sources:

Forge run R-0001

  Task         T-0001  Implement the `median` function in src/lib.rs …
  Agent        claude (claude-code)
  Base commit  83469ef
  Branch       forge/R-0001
  Workspace    .forge/worktrees/R-0001 (removed)

Agent execution
  Status     completed
  Duration   18s
  Exit code  0
  Tokens     164,094 (163,034 in / 1,060 out)
  Cost       $0.1367

Patch
  2 files changed, 20 lines (+19 / -1)
  .forge/runs/R-0001/patch.diff
  committed as 37f89c4

Evaluation integrity
  clean

Evaluation (run by Forge, not by the agent)
  EVALUATOR  CATEGORY  POLICY    RESULT  EXECUTION
  tests      test      required  PASS    completed  exit 0, 97ms
  lint       lint      required  PASS    completed  exit 0, 125ms

  Required          2
  Optional          0
  Metrics           2
  Execution errors  0

Overall
  PASS

Three statuses, never one

A run records what the pipeline did, what the process did, and what Forge concluded — separately. They diverge in ways that matter:

Agent exited	Candidate patch	Checks	Integrity	Outcome
non-zero	present	pass	clean	PASS
zero	present	fail	clean	FAIL
timed out	present	pass	clean	PASS
zero	empty	pass	clean	NO CHANGE
zero	present	pass	protected test deleted	INCONCLUSIVE
zero	present	none configured	clean	INCONCLUSIVE
could not start	—	—	—	ERROR

An unchanged repository passes its own tests trivially, so producing no change is never a pass. Equally, an agent that crashed after writing a correct patch is not penalized for crashing — Forge judges the artifact, not the process.

---

Where run data lives

Everything is under .forge/ in the repository:

.forge/
├── config.toml          # configuration          (commit this)
├── tasks/               # task definitions       (commit these)
├── forge.db             # the experience ledger  (ignored)
├── worktrees/<run-id>/  # agent workspaces       (ignored)
├── runs/<run-id>/       # per-run artifacts      (ignored)
│   ├── prompt.txt           # exactly what the agent was asked
│   ├── agent.stdout.log     # captured agent output
│   ├── agent.stderr.log
│   ├── patch.diff           # the change, read out of Git
│   └── checks/<name>.log    # full output of each check
└── teams/<team-id>/     # final team artifacts   (ignored)
    └── final.patch.diff

forge init writes .forge/.gitignore so run output never enters your history while configuration and tasks do.

Each run also leaves a branch, forge/<run-id>, holding the agent's work as a commit. The workspace directory is removed after a clean run; the branch is not, so the change is always recoverable:

git diff main..forge/R-0001

The ledger is plain SQLite. Prefer the typed CLI query surface for stable automation:

forge history --repository distributed-runtime
forge export --format jsonl

Direct inspection remains possible:

sqlite3 .forge/forge.db "SELECT run_id, status, agent_status, outcome, cost_usd FROM runs"

Evaluator results are normalized too:

sqlite3 .forge/forge.db \
  "SELECT run_id, evaluator_id, kind, required, verdict, execution_status FROM evaluator_results"

Configuration

Historical baseline routing is configured conservatively:

[routing]
minimum_total_evidence = 10
minimum_agent_evidence = 3
minimum_score_margin = 0.05
exploration_policy = "compete_when_uncertain"
periodic_competition_interval = 10

[routing.baseline]
prior_alpha = 1.0
prior_beta = 1.0

The initial team scheduler is deliberately sequential:

[team]
max_parallel_nodes = 1
stop_on_required_node_failure = false

Repository world-model extraction is static and works without an agent:

[world_model]
enabled = true
structure = true
task_metadata = true
history = true

[agents.claude]
executable = "claude"          # for a non-standard install
model = "opus"
timeout_secs = 1800
permission_mode = "acceptEdits"

[agents.codex]
executable = "codex"
model = "gpt-5-codex"
timeout_secs = 1800
sandbox_mode = "workspace-write"
approval_policy = "never"
extra_args = ["--ephemeral"]

The inspected Codex command, JSONL metadata contract, and security mapping are documented in docs/codex-cli.md.

executable, model, timeout_secs, extra_args, and execution_provenance are typed Forge settings. Other keys under [agents.<id>] are passed to that adapter unchanged.

Normal CLI executions are explicitly recorded as live. Deterministic local stub configurations must declare execution_provenance = "synthetic"; older database rows migrate to unknown, never guessed live.

---

Security limitations

Read this before pointing Forge at anything you care about.

A Git worktree alone does not contain a process. Development mode containment.mode = "none" remains available and is visibly unsafe. Production configurations use fail-closed mode = "required", which runs agent and evaluator commands inside a resource-limited Docker-compatible OCI boundary without mounting the user's home. If the runtime, pinned image, network, or declared credential is unavailable, Forge refuses to run rather than falling back to the host. Every run report states the actual posture.

Claude Code runs with bypassPermissions by default. An unattended agent cannot answer a permission prompt, and anything stricter leaves it unable to run the build and test commands its instructions ask for. Set permission_mode = "acceptEdits" under [agents.claude] to tighten it, at the cost of the agent being unable to run commands.

Native/development Codex runs with its workspace-write sandbox and never approval policy by default. This is intentionally not reported as Forge host containment. Under required production containment, Forge's hardened OCI container is the authoritative process boundary and Codex uses its documented externally-contained mode rather than trying to start a nested Bubblewrap namespace. The report makes that distinction explicit as inner sandbox=bypassed, boundary=Forge OCI.

Consequently: use host mode only for development tasks you would run by hand. Supervised production requires the container mode and the operational gates in docs/production-readiness.md.

What Forge does guarantee, with tests:

• It only ever creates or destroys directories inside its configured worktree root, and rejects any run or check name that could escape it.
• Credentials are selected per invocation. A contained agent may request only one supported variable allowed by the profile; production wrappers move the value into private ephemeral authentication state and remove it before model-directed tools run. Captured output is redacted using the exact value, including short credentials.
• Evaluation commands request no provider credential and run as separate conservative contained processes. Before staging or durable capture, Forge scans candidate paths and tracked, untracked, ignored, binary, and symlink content for exact invocation credential values; a match forces destruction.
• Protected evaluation inputs are compared with the recorded base commit; additions, modifications, deletions, and task-scoped exceptions are persisted.
• Ignored build output, Forge runtime files, Git internals, and oversized files are not candidate patch content. Binary additions remain visible but carry a structured warning.
• Anything written outside the workspace is not captured in the patch and never credits the run.

---

Architecture

forge-cli ── forge-runner (execution pipeline)
             ├── forge-agent      provider adapters
             ├── forge-eval       evaluation trust boundary
             ├── forge-executor   process/workspace isolation
             ├── forge-store      experience ledger
             └── forge-git        repository mechanics

forge-router (historical-baseline-v1 selection and explanation)
└── forge-store          trusted routing evidence query

forge-team (validated DAG scheduler)
├── forge-runner         ordinary agent-backed node runs
├── forge-router         automatic node assignment
├── forge-git            commit inheritance and final worktree
├── forge-eval           independent final evaluation
└── forge-store          team plan, lineage, artifacts, and events

forge-world (provider-neutral extraction pipeline)
├── static Rust workspace structure and dependency extraction
├── task/component and immutable ledger evidence extraction
├── commit binding, validation, staleness, and snapshot diff
└── forge-store          immutable snapshots, facts, links, and lifecycle events

forge-policy (policy-baseline-v1 optimization and lifecycle gates)
├── forge-store          cutoff-safe evidence and immutable policy records
├── forge-core           policy, proposal, decision, experiment, and event contracts
└── forge-runner         ordinary execution linkage and bounded policy application

All provider-agnostic domain contracts converge on forge-core.

Crate	Responsibility
forge-core	The vocabulary. No agent, no database, no execution.
forge-git	Repositories, worktrees, diffs.
forge-executor	Process execution, environment policy, workspace provisioning.
forge-agent	The AgentAdapter interface, shared prompt contract, and provider adapters.
forge-eval	Independent evaluation — the trust boundary.
forge-store	The SQLite experience ledger.
forge-router	Candidate resolution, trusted evidence, and the versioned historical baseline.
forge-runner	The run pipeline. The engine a CLI, API, or scheduler each drives.
forge-team	Typed planning and deterministic DAG coordination over ordinary runs.
forge-world	Static, language-extensible extraction into immutable repository snapshots.
forge-health	Immutable repository health snapshots, comparison, and trends.
forge-policy	Cutoff-safe evidence resolution, deterministic optimization, and lifecycle gates.
forge-cli	The forge binary.

Everything provider-specific lives in its adapter file: claude.rs and codex.rs. The prompt is built by one shared function with no agent parameter — two agents given different instructions could not be meaningfully compared.

Invariants worth knowing

These are enforced in code and covered by tests, not just documented.

Forge never trusts an agent's account of its own work. The patch is read from Git; the verdict comes from commands the repository declared in advance. What the agent claimed is recorded as trajectory data and consulted by nothing — including Claude's own is_error flag, which is stored as metadata and never becomes a status.

Green checks require intact evaluation inputs. The trusted task definition is loaded before agent execution. Protected paths are compared against the base commit, and a green check cannot produce PASS after an unapproved protected file addition, modification, or deletion.

A workspace delta is not automatically a candidate patch. Forge respects Git ignore rules, excludes Forge-owned and oversized artifacts with recorded reasons, flags binary additions, and commits only the policy-approved candidate to the durable run branch.

Missing evidence is not a pass. An evaluator that could not execute is recorded as an execution error with an Inconclusive result, distinct from a tool that executed and returned Fail. Required evaluators determine the overall verdict; optional evaluator results remain visible but do not block a pass. Existing task files default every evaluator to required.

Raw measurements are never discarded. Evaluations store raw metrics in their original units alongside normalized dimensions, and there is deliberately no single overall score: weightings will change as evidence accumulates, and they should be recomputable from history rather than requiring re-runs.

Trajectories, not outcomes. Runs are recorded as ordered event streams, because that is the raw dataset a routing model will eventually learn from. Forge records only events it can actually observe — it does not fabricate fine-grained FileRead/FileModified events that the agent interface does not expose. Evaluation lifecycle events identify their real typed subject: an ordinary run or a team execution's integrated final candidate.

---

Development

cargo test --workspace

cargo clippy --workspace --all-targets -- -D warnings

The test suite never invokes a real model or touches the network. Pipeline tests drive a fake AgentAdapter; CLI tests drive the real binary and real adapters against local stub executables.

To smoke-test against the real Claude Code:

./fixtures/new-fixture-repo.sh median /tmp/median && cd /tmp/median && forge init

forge run task.yaml --agent claude

The same controlled fixture can be run with Codex:

forge run task.yaml --agent codex

Or as one controlled infrastructure experiment from an identical base:

forge compete task.yaml --agents claude,codex

Or execute an explicit team plan embedded in the task file:

forge team task.yaml

The fixture ships four failing tests and an unimplemented function, so a PASS requires the agent to have done real work. Its task declares test, lint, security, complexity, benchmark, and custom evaluators. The automated pipeline suite runs the same six-dimensional shape through a deterministic adapter, then changes only security to verify that just that evaluator flips to FAIL.