Add evidence-backed completion reports for agent tasks

[Kelsiito]

What feature would you like to see?

When Codex reports that a task is complete, it should provide a compact, structured mapping between the user's requirements and the observable evidence supporting each completion claim.

Codex currently often ends with a natural-language summary such as "implemented", "fixed", or "all tests pass". That can be useful, but it does not always distinguish clearly between:

• code that was changed;
• behaviour that was directly verified;
• behaviour inferred from inspection;
• checks that were not run;
• requirements that remain partially satisfied or uncertain.

The result is a trust problem: users must reconstruct the evidence manually from a long transcript, and an agent can sound finished even when some acceptance criteria were not tested.

Proposed behaviour

At task completion, Codex should build a lightweight completion record that maps each material requirement to:

• status: verified, implemented but unverified, partially satisfied, blocked, or not addressed;
• relevant files or symbols changed;
• validation commands that were actually run;
• observed results, including failures or skipped checks;
• remaining uncertainty or assumptions;
• optional commit or diff references where available.

Example:

Completion evidence

1. Add password-reset endpoint
   Status: verified
   Changes: src/routes/password_reset.ts, src/services/tokens.ts
   Evidence: npm test -- password-reset -> 12 passed

2. Reject expired tokens
   Status: verified
   Evidence: test `rejects_expired_reset_token` passed

3. Preserve existing login behaviour
   Status: partially verified
   Evidence: focused auth suite passed
   Not run: full end-to-end suite

4. Update API documentation
   Status: implemented, not rendered
   Changes: docs/api.md

The exact representation can be determined by the maintainers and should remain concise by default.

Important distinction

This feature should not expose hidden chain-of-thought or require the model to narrate private reasoning. It should report only externally inspectable evidence: requirements, file changes, commands, results, artifacts, and explicit uncertainty.

Suggested behaviour

• Capture the user's explicit requirements and acceptance criteria at task start or as they evolve.
• Update requirement status as work progresses.
• Record validation commands and their actual exit/result state rather than relying on memory.
• Do not mark a requirement as verified solely because code was written.
• Distinguish focused tests from broad regression coverage.
• Mark stale evidence when relevant files or dependencies change after a test.
• Include failed, skipped, unavailable, or cancelled checks.
• Avoid claiming that all tests pass when only a subset was run.
• Let users expand the report for details or keep it collapsed in normal workflows.
• Persist the completion record across compaction, resume, fork, and handoff.

Why this is useful

This would improve:

• trust in autonomous and long-running work;
• code-review handoffs;
• visibility into incomplete acceptance criteria;
• recovery after compaction;
• multi-agent synthesis;
• auditability in CI, enterprise, and regulated environments;
• the ability to continue a partially completed task without rereading the entire conversation.

It would also reduce false completion reports caused by confusing implementation with verification.

Suggested acceptance criteria

• A task with multiple explicit requirements produces one status entry per material requirement.
• Codex cannot label an item verified without attaching observable validation evidence.
• Tests that were not run are stated as not run.
• A failing check remains visible in the final report even if unrelated checks pass.
• Focused and full-suite validation are distinguished.
• Evidence remains available after context compaction and thread resume.
• The report can be consumed consistently by CLI, Desktop, VS Code, app-server, and subagent orchestration.
• Machine-readable output is available for integrations, while the normal user view remains concise.

Potential implementation direction

Codex already tracks tool calls, file edits, command results, plans, turns, and task state. A completion-evidence layer could aggregate those existing events into a durable, structured record instead of asking the model to reconstruct the entire audit trail from prose at the end.

[leadingproblemsolver]

A completion report gets much more useful if the evidence entries are event-sourced and freshness-aware, rather than reconstructed from the final transcript.

I would give each material claim a small durable record as the work happens:

{
  "requirement_id": "reject-expired-token",
  "status": "verified",
  "claim": "expired reset tokens are rejected",
  "evidence": [
    {
      "kind": "test_result",
      "command": "npm test -- password-reset",
      "exit_code": 0,
      "observed_at": "...",
      "depends_on": [
        {"path": "src/services/tokens.ts", "sha256": "..."},
        {"path": "tests/password_reset.test.ts", "sha256": "..."}
      ]
    }
  ]
}

The important extra rule is evidence invalidation. If src/services/tokens.ts changes after that test ran, the requirement should automatically move from verified to something like implemented_unverified / evidence_stale until the relevant validation is rerun.

That prevents a common false-completion shape:

test passes
→ later repair edits code
→ final summary remembers “test passed”
→ stale evidence is presented as current verification

The report can therefore be a projection over durable facts rather than a model-generated retrospective:

requirements
× observed edits
× validation results
× current source hashes
→ completion projection

A few invariants seem worth making explicit:

• verified cannot be assigned from assistant prose;
• a command result retains exit code and timestamp;
• cancelled/interrupted checks remain visible;
• evidence declares the source/artifact versions it validated;
• edits invalidate only dependent evidence, not the entire report;
• compaction/resume consumes the durable projection rather than regenerating it from memory.

I maintain a small local-first system built around this exact evidence boundary: deterministic decision/blocker/action records, source path + line/content hashes, SQLite persistence, timelines, and source-linked context packs: https://github.com/leadingproblemsolver/living-context-engine

It is not a Codex completion reporter, but it demonstrates the underlying primitive: derived context/reporting remains traceable to versioned source evidence. A Codex implementation could apply the same contract directly to requirements + tool events + repository state.

[Kelsiito]

This is a strong addition. I especially like the evidence invalidation rule — a passing test should only remain valid against the source/dependency state it actually verified.

That also connects nicely with #36717: workspace drift could be one of the mechanisms that marks previously captured evidence as stale when relevant files, dependencies, branch state, or generated artifacts change.

I think that gives the two features a useful separation of concerns:

• Detect workspace drift before continuing stale plans #36717 determines whether the workspace assumptions have drifted;
• this issue determines whether completion claims still have valid evidence.

Making the completion report a projection over durable tool/results state rather than a retrospective model summary also seems much more robust across compaction and resumed sessions.

Thanks for expanding the idea.