Skip to content

[FEATURE] Add reliability, autonomy, and regression reporting #191

Description

@Joncallim

Problem Statement

The contracts and evidence in Epic #184 will not be operationally useful if operators cannot understand why a capability is trusted, restricted, regressing, or revoked. A single letter grade or opaque percentage would hide scope, evidence quality, and critical failures.

Parent Epic: #184
Depends on: #186, #187, #188, #189, #190

Desired Outcome

Forge provides an evidence-first operator surface and exportable reports for project verification goals, capability reliability, autonomy decisions, active Sentinel findings, and recent regressions.

Every summary must link back to the underlying attempts, verification runs, policy evaluations, and evidence.

User Story

As a Forge operator,
I want to inspect what still works, what has earned autonomy, and why anything was promoted or revoked,
So that I can make informed decisions instead of trusting an opaque score.

Requirements

  • Add read APIs/selectors and a minimal operator-facing surface for:
    • active and recently resolved Sentinel findings;
    • verification goals with current status, last green, first observed failure, and pass/fail streaks;
    • capability reliability cohorts and metrics;
    • current autonomy level and operator ceiling;
    • latest promotion/hold/demotion/revocation decision;
    • critical failures, human overrides, and evidence freshness;
    • links to tasks, work packages, runs, artifacts, GitHub issues/PRs, and proof runs.
  • Show scope explicitly: project, capability, resource scope, runtime/model, harness/policy version.
  • Never collapse materially different cohorts into one misleading score.
  • Surface minimum sample size and confidence limitations in plain English.
  • Display critical failures and unresolved verification gaps prominently even when aggregate performance is high.
  • Explain each autonomy decision using stable reason codes and evidence links.
  • Support filtering by project, capability, status, severity, runtime/model, and date window.
  • Provide a machine-readable export suitable for audits or future dashboard/workspace integration.
  • Keep write actions separate and explicit:
    • acknowledge/suppress finding;
    • cap/revoke autonomy;
    • request proof run;
    • request re-verification.
  • Add empty, stale-data, partial-history, and degraded-source states.
  • Add accessibility and responsive checks for any UI added.

Acceptance Criteria

  • An operator can see every project verification goal and its current evidence-backed status.
  • Reliability views identify the exact cohort and sample size.
  • Autonomy views show current level, ceiling, decision reason, policy version, and evidence.
  • Critical failures cannot be hidden by a high aggregate pass rate.
  • Active and resolved Sentinel findings are distinguishable and traceable.
  • Every metric/decision can link to underlying attempts, verifier evidence, or policy evaluation.
  • Operators can cap or revoke autonomy through an explicit audited action.
  • Machine-readable exports preserve reason codes and evidence references.
  • Tests cover filtering, cohort separation, stale/partial data, critical-failure prominence, and permissions.

Out of Scope

  • Replacing the general task/workforce dashboard.
  • An unexplained global A–F grade.
  • Editing policy thresholds without appropriate validation/audit.
  • Automatically resolving findings or restoring autonomy from the reporting view.
  • Building the full Forge Workspace shell before this evidence surface is useful.

Implementation Scope

Medium to Large — queries/APIs, minimal UI, exports, audited operator actions, and tests.

Technical Notes

A useful first panel can be intentionally compact:

What still works
✓ Login                 21 consecutive verified passes
✓ Checkout flow         18 consecutive verified passes
✕ Invoice export        Failed at <commit>; autonomy revoked

Earned autonomy
L3  docs.update          Verified 24/25, scope: docs/**
L1  database.migration   Human approval always required

The display may use friendly labels, but the underlying decisions must remain deterministic and reconstructable from #186/#189 evidence.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions