Skip to content

fix(orchestration): ground whole-plan verify_plan() against DAG-wide tool trace - #6299

Merged
bug-ops merged 1 commit into
mainfrom
fix/6287-whole-plan-verify-grounding
Jul 14, 2026
Merged

fix(orchestration): ground whole-plan verify_plan() against DAG-wide tool trace#6299
bug-ops merged 1 commit into
mainfrom
fix/6287-whole-plan-verify-grounding

Conversation

@bug-ops

@bug-ops bug-ops commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Summary

PlanVerifier::verify_plan() (whole-plan completeness verification, run once after all DAG tasks complete) had no tool-call grounding. Only the per-task verify() path gained deterministic grounding against the real tool-execution trace in #6278/PR #6286, which explicitly scoped whole-plan out with a TODO(critic) marker. A hallucinated aggregated-output claim (e.g. "ran the full test suite across all tasks") could pass whole-plan verification ungrounded.

  • verify_plan() now grounds against the DAG-wide union of every completed task's real tool_trace, rebuilt from transcripts at whole-plan-verify time (independent of whether per-task Verify ran — deliberate defense-in-depth against a future dispatch mode that skips it).
  • Trace availability is all-or-nothing at the DAG level: the aggregate is Some(union) only if every completed task's trace resolves; any one unavailable trace (e.g. a RunInline task) degrades the whole aggregate to None and grounding fails open, exactly reproducing prior ungrounded behavior (now with a debug log noting the degradation).
  • Whole-plan grounding is strictly weaker than per-task grounding at catching a single task's own hallucination (a claim grounds if any task in the plan really performed it) — it is additive defense-in-depth, not a replacement. This is stated explicitly in the updated spec.
  • The trace-path resolution loop is offloaded to spawn_blocking to avoid synchronous transcript reads blocking the async finalization path.
  • specs/009-orchestration/spec.md updated with the new grounding contract, Key Invariant, and AC-13..AC-16.

Also fixes (unrelated, surfaced by the new end-to-end test)

execute_partial_replan_dag (pre-existing whole-plan replan execution code) built a standalone TaskGraph from replan_from_plan's gap tasks and passed it to DagScheduler::new, which enforces 0-based positional task IDs. But replan_from_plan assigns gap-task IDs continuing the parent graph's numbering — always >= 1 in practice, since whole-plan verify only ever runs after tasks complete. Every whole-plan replan attempt was therefore silently rejected by dag::validate and fail-opened to no replan; the pipeline appears to have never actually worked prior to this fix. Gap-task IDs are now remapped to local 0-based IDs for the partial scheduler run and back to the original global IDs on the way out.

Process

Built via /rust-agents:team-develop (new-feature chain): architect → critic (design approved with changes, addressed) → developer → parallel validation (tester, perf, security, impl-critic — all approved, no blockers) → code review (changes requested: one wiring-gap test, one doc-accuracy fix) → developer fix pass (also surfaced and fixed the execute_partial_replan_dag bug, confirmed with the user to keep bundled since narrowing the test would defeat its own purpose) → re-review (approved).

Closes #6287

Test plan

  • cargo +nightly fmt --check
  • cargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warnings
  • cargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler" --lib --bins — 13710 passed, 0 failed
  • RUSTFLAGS="-D warnings" RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" cargo doc --no-deps --workspace --features "desktop,ide,server,chat,pdf,scheduler"
  • gitleaks protect --staged --no-banner --redact
  • LLM serialization gate: live round-trip against local Ollama (qwen2.5:7b) confirming the new claimed_executions/trace-section prompt schema round-trips without error
  • Two new end-to-end integration tests exercising run_whole_plan_verify directly (hallucinated-claim → replan triggered; honest-claim → no replan), proving the full union → verify_plan → grounding → should_replan → replan_from_plan → execute_partial_replan_dag pipeline is genuinely wired
  • Unit test guarding the empty-but-available-trace pitfall (Some(vec![]) must never collapse to None)
  • Full /plan-CLI-level live testing (multi-task plan → whole-plan verify → grounding fires) — documented as Scenario G in .local/testing/playbooks/orchestration-verifier-grounding.md, left for a future CI live-testing cycle per project convention (CI sessions are separate from implementation sessions)

@github-actions github-actions Bot added bug Something isn't working documentation Improvements or additions to documentation rust Rust code changes core zeph-core crate size/XL Extra large PR (500+ lines) and removed bug Something isn't working labels Jul 14, 2026
@bug-ops
bug-ops enabled auto-merge (squash) July 14, 2026 20:54
@bug-ops
bug-ops force-pushed the fix/6287-whole-plan-verify-grounding branch from 1747a95 to a68c7cc Compare July 14, 2026 20:54
@github-actions github-actions Bot added the bug Something isn't working label Jul 14, 2026
…tool trace

PlanVerifier::verify_plan() (whole-plan completeness verification, run once
after all DAG tasks complete) had no tool-call grounding. Only the per-task
verify() path gained deterministic grounding against the real tool-execution
trace in #6278/PR #6286, which explicitly scoped whole-plan out with a
TODO(critic) marker. A hallucinated aggregated-output claim could pass
whole-plan verification ungrounded.

verify_plan() now grounds against the DAG-wide union of every completed
task's real tool_trace, rebuilt from transcripts at whole-plan-verify time
by reimplementing the resolution logic build_tool_trace_for_task uses for
the per-task path (independent of whether per-task Verify ran for a given
task -- deliberate defense-in-depth against a future dispatch mode that
skips it). Trace availability is all-or-nothing at the DAG level: the
aggregate is Some(union) only if every completed task's trace resolves; any
one unavailable trace degrades the whole aggregate to None and grounding
fails open, exactly reproducing prior ungrounded behavior. Whole-plan
grounding is strictly weaker than per-task grounding at catching a single
task's own hallucination -- it is additive defense-in-depth, not a
replacement. The trace-path resolution loop is offloaded to spawn_blocking
to avoid synchronous transcript reads blocking the async finalization path.

Also fixes a pre-existing, previously-untested bug in
execute_partial_replan_dag (whole-plan replan execution, unrelated to
grounding) surfaced by the new end-to-end test: gap tasks from
replan_from_plan carry IDs continuing the parent graph's numbering, but the
standalone partial TaskGraph built to execute them requires 0-based
positional IDs, so every whole-plan replan attempt against a non-empty
graph was silently rejected by dag::validate and fail-opened to no replan.
Gap-task IDs are now remapped to local 0-based IDs for the partial
scheduler run and back to the original global IDs on the way out.

specs/009-orchestration/spec.md updated with the new grounding contract,
Key Invariant, and AC-13..AC-16.

Closes #6287
@bug-ops
bug-ops force-pushed the fix/6287-whole-plan-verify-grounding branch from a68c7cc to 337446b Compare July 14, 2026 20:55
@bug-ops
bug-ops merged commit f1d6ace into main Jul 14, 2026
43 checks passed
@bug-ops
bug-ops deleted the fix/6287-whole-plan-verify-grounding branch July 14, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working core zeph-core crate documentation Improvements or additions to documentation rust Rust code changes size/XL Extra large PR (500+ lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

orchestration: whole-plan verify_plan() aggregated-output verification remains ungrounded

1 participant