Skip to content

feat(manifest): run manifest coverage contract for review (#367) - #520

Merged
lizhengfeng101 merged 14 commits into
alibaba:mainfrom
Gongyl01:367-run-manifest
Aug 1, 2026
Merged

feat(manifest): run manifest coverage contract for review (#367)#520
lizhengfeng101 merged 14 commits into
alibaba:mainfrom
Gongyl01:367-run-manifest

Conversation

@Gongyl01

@Gongyl01 Gongyl01 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #367.

Adds a versioned, immutable RunManifest to ocr review that records per-item coverage (selected / completed / reused / failed / waived) and an authoritative terminal state (complete / partial / failed / skipped). On successful persistence, CLI JSON and session_end.run_manifest expose the same frozen manifest, so a partial review can no longer be mistaken for ordinary success or a complete no-findings result.

Built on top of PR #306's resumable sessions; this is a narrow output-contract follow-up, not a parallel persistence system.

  • New internal/session/manifest.go: RunManifest value object (schema ocr.run-manifest/v1), ManifestBuilder (single state table, unified transition entry, RegisterSelected / MarkCompleted / MarkReused / MarkFailed / MarkWaived), SealSelected (closes the selected denominator before dispatch), Finalize(elapsed) (RunManifest, error) with hard validation, structured RunFailure + RunFailureClass enum, and sanitizeReason redaction floor.
  • internal/agent/agent.go: a pre-dispatch pass (registerCoverage) registers all post-filter, non-deleted items and seals them before resume reuse or goroutine dispatch; markCompleted / markReused / markFailed record per-item outcomes. The live --max-tokens-budget gate records FailureBudget as the pending-item failure cause, so only undispatched/pending items are swept to failed(budget); BudgetExceeded() continues to feed the summary and warning. SetRunFailure is reserved for run-level stop causes at the trigger source (input failure on loadDiffs, internal failure on registerCoverage). initManifest initializes requested input and execution metadata; resolved input and repository identity are captured from the actual run and attached before finalization.
  • internal/diff/git.go: ResolveInput freezes resolved base/head/exact range per input mode, including first-parent merge input to match GetDiff --diff-merges=first-parent; RemoteIdentity / canonicalRemote produce a credential-free repository identity by canonicalizing the origin URL to host[:port]/path (dropping userinfo/query/fragment) and SHA-256ing the result; local remotes omit identity.
  • internal/session/history.go: NewManifestBuilder mounts the builder on SessionHistory; Finalize now returns error (was void) and uses sync.Once so session_end is written exactly once while replaying the first error on every call.
  • internal/session/persist.go: WriteSessionEnd writes session_end as the last physical JSONL record, embeds the frozen manifest under run_manifest, and surfaces write/flush/close errors.
  • cmd/opencodereview: review JSON exposes manifest; top-level status uses the four terminal states (scan stays legacy); aggregate-budget reporting remains available through summary.budget_exceeded; failed runs include the sanitized run-failure cause or failed/selected counts; text output no longer shows Looks good to me. for partial/failed; session list/show and the viewer prefer a valid session_end.run_manifest and otherwise display legacy/unknown state without faking complete.

Design (key invariants)

  • Single source of truth. The terminal state is computed once in Finalize. On successful persistence, CLI JSON and session_end.run_manifest serialize the same frozen manifest value. JSON output no longer derives review status from warnings.
  • Single outcome entry. All per-item outcomes go through one transition function. Registration after seal, transitions after freeze, unknown item IDs, invalid failure classes, empty waiver reasons, and conflicting transitions return errors. Repeating the same outcome is idempotent; re-marking a failed item with a different classification is rejected.
  • Seal before dispatch. SealSelected closes the selected denominator after the pre-dispatch pass; resume-reused and to-be-dispatched items enter the same frozen set, so selected = completed ∪ reused ∪ failed ∪ waived always holds.
  • Explicit stop cause. run_failure is recorded at the trigger source (input = diff resolution, internal = scheduler/invariant). It is never inferred from ctx.Err(). Ordinary aggregate budget exhaustion is a controlled coverage stop wired to --max-tokens-budget: it sets a pending FailureBudget cause and sweeps only uncovered items, without creating run_failure. Contract-level recorders for cancelled and run-level timeout, plus run_failure.classification=budget for a genuine counter/scheduler anomaly, remain available but have no live trigger in this release (no new SIGINT handler or global deadline).
  • Coverage ≠ findings. Findings count never participates in terminal-state computation; complete may have zero or many findings.
  • Manifest-safe redaction. Production manifest failure reasons use fixed, allow-listed summaries instead of raw provider errors. sanitizeReason provides a second redaction floor for URL credentials, Bearer/Basic tokens, credential-like assignments, and control characters. CLI output suppresses raw subtask_error warnings when a manifest is present. Existing session checkpoint and conversation persistence behavior is unchanged.
  • Persistence errors surface. Finalize / WriteSessionEnd return errors up the stack across all exit paths—the no-files path, the loadDiffs-failure path, the normal dispatch path, and the scan path. On the normal path, dispatch and persistence errors are reported via errors.Join. A persistence failure does not rewrite the frozen manifest; it surfaces as a delivery error with a non-zero exit code.
  • Resume preserves lineage, not copied input. A child run records its own currently resolved input and links the direct parent through parent_run_id; immutable-ref drift rejection is unchanged and remains out of scope.

Terminal state

State Condition
complete selected non-empty, failed empty, no run_failure
partial 0 < failed < selected, no run_failure
failed all selected failed, or run_failure present
skipped selected empty, no run_failure

waived items count as covered, so a run containing only completed / reused / waived items is complete. Findings count never affects the state.

scan is unchanged and continues to emit the legacy status values; the two commands do not yet share the same status contract.

Compatibility

  • ocr review --format json intentionally migrates top-level status from the legacy success / completed_with_errors / completed_with_warnings values (and feat(agent): add token/tool-call cost guardrails to the review path #508's budget_exceeded) to the manifest terminal states complete / partial / failed / skipped.
  • Budget-aware consumers should read summary.budget_exceeded or inspect manifest.coverage.failed[].classification == "budget"; the terminal state remains coverage-derived instead of being overwritten by the budget flag.
  • A budget stop exits 0 when any selected item was covered (partial), and exits non-zero when every selected item failed (failed). scan retains its legacy output contract.

Upstream integration

This branch is synchronized through upstream/main@c391892 and preserves the upstream behavior merged while #367 was in progress:

  • feat(agent): add token/tool-call cost guardrails to the review path #508 aggregate token budget: keeps the flag, dispatch guardrail, usage summary, warnings, and BudgetExceeded() signal; its former status=budget_exceeded output is adapted to the manifest terminal-state contract described above.
  • fix: honor per-file review terminal states #582 per-file completion semantics: only task_done(status=DONE) marks an item completed; FAILED, provider errors, missing completion, and structured MainLoopStop reasons keep the item failed and flow into manifest coverage.
  • Merge conflicts in CLI output, agent scheduling/budget handling, llmloop termination, and scan call sites were reconciled to retain both upstream behavior and the manifest's single-source-of-truth invariants.

How to test

# 1. Run a review; inspect the manifest in JSON output.
ocr review --from main --to feature --format json | jq .manifest

# 2. A mixed-success run shows terminal_state=partial and lists failed items.
#    Text mode no longer prints "Looks good to me." for partial/failed.

# 3. Resume records parent_run_id and marks checkpoint hits as reused.
ocr review --from main --to feature --resume <session-id> --format json | jq .manifest.parent_run_id
ocr review --from main --to feature --resume <session-id> --format json | jq .manifest.coverage.reused

# 4. session list/show and viewer prefer session_end.run_manifest.
ocr session list
ocr session show <session-id>
ocr viewer

Checklist

  • go test -race -count=1 ./... green
  • go vet ./... clean
  • gofmt -l and go mod tidy produce no diff
  • go build ./... succeeds; CLI smoke checks pass
  • GitHub Actions and CLA checks are green
  • Tests cover the implemented v1 verification matrix: full success, zero findings, mixed failures, all-failed, aggregate-budget partial/all-failed exit boundaries, run-level failure (input / internal), per-item timeout/budget/panic, skipped, resume (parent_run_id + reused), provider transition, task_done completion semantics, structured main-loop stops (max_rounds / empty_rounds / compression), cancellation contract, SealSelected lifecycle, failed-finalize rollback, conflicting/idempotent transitions, invalid failure class, empty waiver reason, cross-exit manifest consistency, malformed/legacy/aborted session display, large JSONL/non-EOF reader behavior, and security redaction
  • On successful persistence, CLI JSON manifest and session_end.run_manifest serialize the same frozen manifest value
  • Manifest fields and review CLI JSON do not expose raw provider errors, credentials, diff/prompt/response bodies, or provider tokens

Out of scope (deliberately)

  • scan is not wired to the v1 manifest (Non-Goal; it passes a nil manifest and emitRunResult is nil-safe) and keeps its legacy status values.
  • Waiver user entry is deferred to a follow-up; this release only ships the waived output semantics.
  • cancelled and run-level timeout have contract coverage but no live source in this release—no new SIGINT handler or global deadline is added. Normal aggregate budget exhaustion is live through --max-tokens-budget and is represented as per-item failed(budget) coverage; run_failure.classification=budget is reserved for a separate run-level counter/scheduler anomaly and has no live trigger.
  • Resume matching and checkpoint persistence are unchanged. A resumed child records its own currently resolved input and links the direct parent through parent_run_id; ref-drift rejection and "diff by immutable SHA" are deferred.
  • Old sessions are displayed as legacy, never faked as v1 complete.

Gongyl01 and others added 9 commits July 20, 2026 15:39
First slice of issue alibaba#367 (run manifest coverage contract): the data
model and state machine only. Not yet wired into the agent or CLI, so
existing review/scan output is unchanged.

Introduce the versioned, immutable RunManifest (schema ocr.run-manifest/v1)
and a concurrency-safe ManifestBuilder that tracks per-file coverage
(selected/completed/reused/failed/waived) and freezes into a terminal
state.

- terminal state derived solely from coverage sets, never comments/warnings
  (complete/partial/failed/skipped)
- Finalize sweeps any undecided selected item to failed/unknown so no item
  is silently dropped
- single-mutex builder: first terminal state wins, frozen after Finalize,
  nil-receiver safe
- fixed failure classification enum with an unknown catch-all
- redaction floor on failure/waive reasons (strip secrets, cap length) as a
  single write entry so callers cannot bypass it
- 22 unit tests, race-clean

Refs: issue alibaba#367
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address findings from the concurrency / JSON-contract / PR#306-coupling
adversarial review of the manifest data model (still slice 1; not wired to
agent or CLI).

- SetSweepClass: Finalize can classify undispatched items as cancelled/budget
  instead of a blanket unknown (the one real model gap the review found)
- ItemID(fingerprint)=SHA-256 canonical mint helper; an item_id is never a raw
  fingerprint, keeping the resume cross-reference explicit and mix-ups caught
- sanitizeReason: strip control/ANSI chars, coerce valid UTF-8, redact quoted
  secret values, guarantee single line
- Finalize returns deep-copied coverage slices so the frozen snapshot is never
  aliased across the two outlets
- RegisterSelected: nil-safe (lazy-init map) + documents that only the
  post-deletion/post-filter dispatchable set may be registered

+7 unit tests (29 total), race-clean.

Refs: issue alibaba#367
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ure (shard ②d)

- Freeze per-mode input identity (mode + resolved_base/head + exact_range +
  source_artifact_sha256) via diff.ResolveInput/commitParents, and repository
  identity via RemoteIdentity/canonicalRemote (credential-free).
- Add rule_config_sha256 and runtime_config_sha256 over an allowlist of
  non-secret fields using a length-prefixed SHA-256 framework (no tokens/URLs).
- Replace SetRunLevelFailure(bool) with structured SetRunFailure(class, reason)
  and set ManifestInput.mode; fill execution.* (ocr version, provider, model,
  concurrency, config hashes).
- Thread error returns through Finalize/WriteSessionEnd (main review path
  surfaces them; skip/all-failed/scan paths hardened in follow-up).
- Tests: manifest_hash, canonical_config, git_resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lassification

Merged review themes A/B/E from the 07-22 consolidated assessment.

Theme A — Finalize / session_end delivery errors no longer swallowed:
- agent.go no-files path returns the Finalize error instead of nil (A1)
- agent.go loadDiffs failure joins the Finalize error via errors.Join (A2)
- session.Finalize uses sync.Once + cached finalizeErr: written exactly
  once, concurrency-safe, and every caller replays the same result so a
  retry cannot falsely report success (A3)
- scan/agent.go wires both Finalize call sites to surface the error (A4)

Theme B — canonicalRemote rewritten (internal/diff/git.go):
- keep the port (u.Host, not u.Hostname) so endpoints differing only by
  port stay distinct (B1)
- split scp syntax on the first ':' so an '@' inside the path survives (B2)
- recognize local/file/Windows/UNC remotes and omit identity rather than
  misparsing a path as a host (B3; local-remote policy still open)

Theme E — main_task-empty is now a sentinel (errMainTaskEmpty) classified
via errors.Is instead of matching error text.

Theme D (TOCTOU) deferred to shard 4 per issue alibaba#367 open-issues OI-12.

Tests: go build ./... + go vet + go test ./... all green (23 pkgs).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mal path

The success-path Finalize wiring used `ferr != nil && err == nil`, so when the
review (or scan) failed AND session_end also failed to persist, the persistence
error was dropped and only the dispatch error surfaced — the caller never
learned the session/manifest was not saved.

Join both with errors.Join when both occur (matching the loadDiffs path), so a
persistence failure is always reported even alongside a dispatch failure. This
closes the last gap in the OI-10 contract.

- internal/agent/agent.go: review normal path
- internal/scan/agent.go: scan normal path (+ errors import)

Tests: go build ./... + go vet + go test ./... all green (23 pkgs).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 使用冻结 manifest 统一 review JSON、文本与退出状态\n- session CLI 和 viewer 展示五集合覆盖并兼容 legacy/aborted\n- 补充本地 mock、跨出口一致性及安全验收用例
验收用例:configuration 分类(run 级 sweep + item 级映射)、budget/timeout/panic 混合 partial 隔离、跨出口一致性改为规范化原始字节比对、flag 校验失败无产物断言。

代码修复:sanitizeReason 先剥控制字符再脱敏(堵控制字节绕过)、失败项异分类二次标记报冲突错误、source_artifact_sha256 按 item_id 去重并稳定排序、sortItems 改 SliceStable 对齐设计用词。

全仓 go test 23 包通过。
覆盖 issue alibaba#367 验收标准 provider transition:resume 时 provider/model 改变后,子 manifest 记录当前值而非继承父运行,并经 parent_run_id 链接父会话以支持审计。用 mock client,不依赖真实 provider key。
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🔍 OpenCodeReview found 7 issue(s) in this PR.

  • ✅ Successfully posted inline: 7 comment(s)

Comment thread cmd/opencodereview/output.go
Comment thread cmd/opencodereview/review_cmd.go
Comment thread cmd/opencodereview/session_cmd.go
Comment thread internal/llmloop/loop.go
Comment thread internal/session/manifest.go
Comment thread internal/viewer/store.go
Comment thread internal/viewer/store.go Outdated
// fixed token ceiling. session_end embeds the complete run manifest and can
// legitimately exceed the former 10 MiB scanner limit on very large reviews.
func readJSONLLines(r io.Reader, visit func([]byte)) error {
reader := bufio.NewReader(r)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance: The stated motivation for this refactor is to handle lines exceeding the former 10 MiB scanner limit. However, bufio.NewReader uses a default 4096-byte buffer. For very large JSONL lines (multi-megabyte manifest records), ReadBytes will repeatedly grow its internal buffer through many small allocations and copies, causing significant GC pressure. Consider using bufio.NewReaderSize with a larger initial buffer to reduce allocation overhead for the expected large-line use case.

Suggestion:

Suggested change
reader := bufio.NewReader(r)
reader := bufio.NewReaderSize(r, 64*1024)

Gongyl01 added 3 commits July 30, 2026 17:05
统一聚合预算停止时的 coverage、status 与退出码。传播 session writer 初始化错误,并补齐 merge first-parent 输入身份及回归测试。移除代码注释中的外部设计文档引用。
# Conflicts:
#	cmd/opencodereview/output.go
#	internal/agent/agent.go
#	internal/llmloop/loop.go
#	internal/scan/agent.go

@lizhengfeng101 lizhengfeng101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge conflicts

冲突解决:
- cmd/opencodereview/flags.go: 接受 upstream 删除(alibaba#625 迁移到 Cobra),
  将 --max-tokens-budget 的说明文案迁移到 shared_flags.go 的 addConcurrencyFlags
- internal/viewer/store.go: SessionSummary 取双方字段并集(manifest 终态 + CommentCount);
  LoadSession 同时保留 CommentCount 统计与 readErr 返回
- internal/viewer/templates/sessions.html: 同时保留 Status 与 Comments 两列

另修复两处自动合并后编译失败:
- store.go ListSessions: 本分支已将 scanner 循环重构为 readJSONLLines 回调,
  alibaba#627 新增的 continue 落入闭包,改为 return
- compat_test.go: 补 runReview 兼容包装(alibaba#625 已拆为 reviewCmd + executeReview)

go build / go vet / go test ./... 全部通过。
@Gongyl01
Gongyl01 requested a review from lizhengfeng101 August 1, 2026 00:31

@lizhengfeng101 lizhengfeng101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lizhengfeng101 lizhengfeng101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lizhengfeng101
lizhengfeng101 merged commit 0ce730a into alibaba:main Aug 1, 2026
7 checks passed
@Gongyl01
Gongyl01 deleted the 367-run-manifest branch August 3, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Emit a complete immutable review manifest and partial-coverage contract

2 participants