[FEATURE] Add capability reliability ledger

[Joncallim]

Problem Statement

Forge cannot safely infer trust from a task-level pass/fail count. Capabilities differ by project, resource scope, runtime, model, policy version, and verification quality. A global agent score would hide material differences and encourage unsafe autonomy.

Parent Epic: #184
Depends on: #185

Desired Outcome

Forge records an append-only, evidence-backed history of comparable capability attempts and calculates transparent reliability metrics without letting the execution worker grade itself.

Reliability is keyed to a bounded cohort such as:

project + capability + scope fingerprint + runtime/model + harness/policy version

User Story

As a Forge operator,
I want to see how reliably a specific capability has performed in a specific scope,
So that trust and autonomy decisions reflect comparable verified evidence rather than a vague agent reputation.

Requirements

• Add a durable capability-attempt contract linked to the canonical outcome from [FEATURE] Normalize execution outcomes and stop reasons #185.
• Include at least:
  • attempt id;
  • project id;
  • task/work-package/agent-run references;
  • capability key;
  • normalized scope fingerprint;
  • runtime/provider/model snapshot;
  • harness/policy version;
  • outcome and stop-reason code;
  • verification result and verifier reference;
  • acceptance-criteria totals;
  • validation-command totals;
  • human override/rejection/rollback flags;
  • evidence references;
  • timestamps.
• Define which attempts are comparable. Material runtime/model, policy, capability, or scope changes must start a new cohort or require requalification.
• Make attempt ingestion idempotent and append-only for auditability.
• Calculate deterministic metrics such as:
  • first-attempt success rate;
  • independently verified pass rate;
  • repair/retry rate;
  • human rejection rate;
  • rollback/regression rate;
  • policy-block rate;
  • false-success rate;
  • consecutive verified passes;
  • evidence freshness.
• Support rolling windows and minimum sample sizes.
• Weight or classify critical failures separately so averages cannot conceal them.
• Exclude self-reported confidence or prose grades from authoritative calculations.
• Add data retention, redaction, and migration notes.

Acceptance Criteria

• Two attempts with materially different capability scopes are not silently combined.
• Runtime/model or policy-version changes are visible and can trigger requalification.
• Every ledger entry links to a canonical outcome and verification/evidence state.
• Reprocessing an attempt is idempotent.
• Metrics can be recomputed deterministically from stored attempts.
• A critical failure remains visible regardless of the aggregate pass percentage.
• Human rejection, rollback, and override events affect the reliability view.
• Missing independent verification is not counted as a verified pass.
• Tests cover cohorting, rolling windows, critical failures, idempotency, and historical data.

Out of Scope

• Granting or revoking autonomy; that belongs to [FEATURE] Add evidence-based earned autonomy policy engine #189.
• A polished dashboard; that belongs to [FEATURE] Add reliability, autonomy, and regression reporting #191.
• One global grade for an agent, model, or workforce.
• Allowing a worker’s self-evaluation to create a passing ledger entry.
• Replacing existing run, artifact, or task history.

Implementation Scope

Large — data model, evidence ingestion, deterministic aggregation, and tests.

Technical Notes

Potential persistence:

• capability_attempts as immutable attempt evidence;
• optional materialized reliability summaries that can always be rebuilt;
• a stable scope fingerprint derived from normalized allowed paths/resources/capabilities, never raw secrets;
• foreign keys to existing project/task/work-package/agent-run records where available.

Use PostgreSQL as authority. Redis may schedule recomputation but must not hold the canonical ledger.

[0xbrainkid]

This is the right level of granularity. A capability reliability ledger is much safer than a global agent score because it keeps the score tied to a comparable scope, runtime, verifier, and policy version.

One field I would make first-class is an exportable receipt envelope for each attempt, separate from the materialized reliability summary:

• agent_id / worker identity
• capability_key and normalized scope fingerprint
• runtime/model/policy/harness versions
• canonical outcome id and verifier identity
• evidence refs or hashes
• human override / rollback / dispute state
• timestamp plus decay or expiry semantics

Then the summary can always be recomputed, and another system can decide whether that evidence applies to its own task. That matters for cross-org agent marketplaces: a buyer should be able to ask "has this agent produced verified receipts for this kind of work under comparable constraints?" without trusting Forge's aggregate number as a broad permission slip.

The AgentFolio/SATP boundary we keep coming back to is similar: identity proves continuity, receipts prove observed outcomes, and local policy decides routing or autonomy. Keeping those layers explicit here would make Forge's ledger easier to consume by registries, marketplaces, and delegation systems without forcing Forge to become the universal reputation backend.

[Joncallim]

Architecture accepted — ready for Backend handoff

Design opened as draft PR #326. Implementation has not started.

• Primary document: docs/architecture/issue-186-capability-reliability-ledger.md
• ADR: docs/adr/0012-capability-reliability-ledger.md

Selected as the next issue because Epic #184's own recommended order puts it here and both prerequisites are on main: #201 (ADR 0011, operation_runs) and #185 (ADR 0010, execution_outcomes).

Decisions worth knowing before implementation starts

• Cohorting makes requalification implicit. A cohort fingerprint covers project, capability key, scope, runtime/model, and policy. A material change to any of them produces a different cohort, so a new sample count starts rather than an old one silently carrying over. The four component fingerprints are stored alongside the cohort so drift is attributable to a specific input, not just detectable.
• One row per (outcome, capability), not per package. Autonomy decisions in [FEATURE] Add evidence-based earned autonomy policy engine #189 are per capability, so attempts fan out — sharing an attempt_group_id and a multiplicity count, so one failed package covering five capabilities is never read as five independent failures. A missing or oversized classification writes one reserved unclassified row so the measurement gap stays visible instead of being guessed at.
• Attempts are immutable; later evidence is appended. Verification, human decisions, rollback, override, and detected drift go to a separate adjudication table in gapless sequence order, enforced by triggers.
• verification_mode decides what counts as verified. self_reported and human_review never contribute to the independently verified pass rate. independent_agent has no producer until [FEATURE] Add independent Verification Workforce execution #188, so v1 refuses it at ingest rather than storing an unbacked value. The current gap is reported as an explicit unverified-completion rate instead of being folded into a pass rate — for most cohorts today, the honest answer is "completed, but nobody independent checked it".
• No free-text column anywhere in the ledger. Every text column is a closed enum, a 64-hex fingerprint, or the bounded capability-key grammar, each enforced by a CHECK. Prose, paths, and credentials cannot enter it even by accident; scope is fingerprinted from the opaque root_ref, never local_path.
• No materialized summary in v1. Metrics are a pure function of stored evidence. §6.3 records the measurement that would justify adding a cache later; a cache that can disagree with its evidence is the failure class this ledger exists to avoid.
• Fails closed. Below minimum samples → insufficient_evidence; a linked outcome that changed since ingest → evidence_drift with all rates suppressed. Critical failures are reported unconditionally in every state.

Two criteria only partially satisfiable here

Called out in §14 rather than claimed:

• "rollback … events affect the reliability view" — storage contract, metric, and tests exist, but no producer emits rollback_recorded until [FEATURE] Add evidence-based earned autonomy policy engine #189/[FEATURE] Add Project Sentinel detection and escalation flow #190. Notably, rework is not rollback; synthesizing one from a needs_rework decision would corrupt the metric [FEATURE] Add evidence-based earned autonomy policy engine #189 depends on.
• "historical data" — covered as "attempts predating the table are readable as missing evidence", not as backfilled rows, consistent with ADR 0010.

Open decision for you

The receipt-envelope suggestion above is recorded in §16 as considered and deferred. The attempt row already carries nearly that field set, so an export projection stays cheap to add — but publishing reliability evidence across a trust boundary implies a stable public schema, portable agent identity, signing, and revocation. That reads as a product decision rather than an implementation detail of this ledger, so it is parked pending your call instead of being absorbed into the slice.

Handoff shape

Nine work packages in §13 (contracts → migration + CI gates → ingest → operation ingest → adjudications → metrics/reader → inspection script → QA matrix → docs). Review requirement both, with Security/Adversarial review applying to the ACL change and the no-free-text invariant. §15 lists the eight stop conditions where the implementer must escalate instead of improvising.

§10 pins what migration 0031 will trip: 31 migrations today with journal max 1785993600000, four files carrying those literals, plus the closed application-ACL inventory in web-ci.yml where both new tables must expect exactly SELECT and INSERT — no UPDATE, no DELETE.

---

Generated by Claude Code