research(orchestration): MAST-informed handoff hardening to reduce multi-agent coordination failures

[bug-ops]

Source

MAST: Why Do Multi-Agent LLM Systems Fail? (arxiv 2503.13657, UC Berkeley March 2025)
https://arxiv.org/abs/2503.13657

Core Finding

Analysis of 1,642 execution traces across 7 multi-agent frameworks found that coordination breakdowns account for 36.9% of all failures -- the single largest failure category. The taxonomy identifies 14 failure modes across three clusters:

1. System design failures -- agent boundaries too coarse, missing capability overlap
2. Inter-agent misalignment -- ambiguous task handoff, missing context propagation, incompatible role assumptions
3. Task verification failures -- no confirmation that subtask output meets specification before continuing

The inter-agent misalignment cluster (most relevant to Zeph) covers:

• Ambiguous handoff: the receiving agent doesn't have enough context to continue correctly
• Missing context: key facts known to the orchestrator are not passed to the sub-agent
• Role assumption mismatch: orchestrator assumes capabilities the sub-agent doesn't have

Current Zeph Gap

Zeph's sub-agent handoff uses YAML files with ad-hoc context fields (id, parent, agent, timestamp, status, context, output, next). The context and output fields are free-form, and there is no schema validation that required context is present before a handoff is issued.

From MAST's taxonomy, the most likely failure modes in Zeph are:

• Missing context: sub-agent receives task description but not the supporting facts (e.g., which crate, which function, what the constraint is)
• Ambiguous task: handoff says "fix the bug" without specifying the exact error or reproduction path
• No verification gate: the orchestrator proceeds to the next phase without confirming the sub-agent's output is valid

Implementation Sketch

1. Define a typed HandoffContext struct with required fields per agent type (architect needs: spec files, constraints; developer needs: spec + architect output; tester needs: test plan + implementation)
2. Add a validation step in SubAgentManager before dispatching: verify required context fields are present
3. Add a verification step after sub-agent completion: check that output fields match expected structure before triggering the next phase
4. Log handoff quality metrics: context completeness score, verification pass/fail rate

This does not require architectural changes to the sub-agent system -- it's a validation layer on top of the existing YAML handoff protocol.

Complexity

Medium. Requires: typed HandoffContext enum per agent role, validation logic in SubAgentManager, and either LLM-based or schema-based output verification. No changes to transport or agent execution.

Expected Benefit

• Reduces the 36.9% coordination failure rate (most impactful single change for multi-agent reliability)
• Makes handoff failures explicit and debuggable instead of silent
• Enables better error recovery: if context is missing, the orchestrator can request it before dispatching

See Also

• SubAgentManager: zeph-subagent crate
• Handoff protocol: .local/handoff/ + rust-agent-handoff skill
• Decentralized task allocation: https://www.nature.com/articles/s41598-025-21709-9
• Multi-agent coordination survey: https://arxiv.org/html/2501.06322v1

[bug-ops]

Implementation Progress (Issue #2023)

✅ Completed

Phase 1 (PR #2076) — HandoffContext Type Vocabulary

• Type definitions for all 6 agent roles (Architect, Developer, Tester, Reviewer, Critic, Generic)
• ValidationRule enum + ValidationResult types
• VerificationStatus enum (Verified/PartiallyVerified/Failed/Unverified)
• Merged: ee51ef6f

Phase 2 (PR #2078) — Schema-Driven Validation Layer

• validate_context() with 4 rules:
  • ObjectiveNonEmpty (Hard: empty → blocked)
  • CriteriaPresent (Hard: ≥1 required)
  • HandoffRefSupported (Hard: rejects ById in Phase 1-2)
  • RoleContextComplete (Hard: per-variant required fields)
• verify_output() + derive_verification_status() — post-completion verification
• DagScheduler::tick() wiring — hard failures block dispatch (task → Failed + propagate_failure)
• HandoffMetrics — dispatch/verification counters, rule_violations, role_counts
• build_task_prompt() update — structured DependencyOutput integration
• Test coverage: 6312 tests (+48 vs Phase 1), all 4 validators approved
• Status: Ready for merge (CI passing after unused import fix)

---

📋 Planned (Phase 3-5)

Phase 3 — Output Verification Gate & LLM-Assisted Criteria

• verify_output() expansion: structured criteria results (currently empty → Unverified)
• LLM-powered criteria extraction from agent output
• HandoffStore (SQLite) for persistent handoff history
• TUI integration: new "Handoff Quality" metrics panel

Phase 4 — Handoff File Migration & Registry

• Migrate handoff files from .local/handoff/ to .zeph/handoff/ (persistent)
• HandoffSchemaRegistry discovery (if extensible roles added in future)
• Coordination with rust-agents plugin for file paths

Phase 5 — Quality Analytics & Dashboarding

• Per-role failure rate analysis (which roles have highest retry counts?)
• Per-rule violation frequency (which validation rules trigger most?)
• /plan handoff-report — historical trend analysis
• TUI orchestration view: show validation/verification status inline with task graph

---

🎯 MAST Taxonomy Compliance

Failure Mode	Phase	Solution
Ambiguous handoff	2	CriteriaPresent enforces ≥1 criterion
Missing context	2	RoleContextComplete validates required fields per role
No verification gate	2	verify_output() validates criteria coverage post-completion
Invalid criteria	3	LLM-powered extraction from structured output
Verification metrics	5	Per-rule/role failure analytics + TUI dashboard

---

📊 Metrics Added (Phase 2)

• handoff_metrics.total_dispatched — all dispatch attempts
• handoff_metrics.clean_dispatches — passed validation
• handoff_metrics.warned_dispatches — soft warnings
• handoff_metrics.blocked_dispatches — hard failures (blocked)
• handoff_metrics.{verified,partial,failed,unverified}_completions — post-completion status
• handoff_metrics.rule_violations — per-rule failure counts
• handoff_metrics.role_counts — per-role dispatch frequency

---

🔗 Related Issues

• research(tools): tool dependency graph for sequential tool availability #2024 — Orchestration metrics TUI integration (Phase 3)
• fix(sanitizer): injection false positives from legitimate user queries in memory retrieval #2025 — Handoff file persistence (.zeph directory layout) (Phase 4)
• feat(tools): dynamic tool schema filtering to reduce context waste and improve selection accuracy (#2020) #2026 — Quality analytics dashboard (Phase 5)

---

Next Steps

1. Merge PR feat(orchestration): implement Phase 2 handoff validation layer (issue #2023) #2078 once CI passes
2. Update CHANGELOG.md with Phase 2 release notes
3. Plan Phase 3 scope (structured output parsing + LLM criteria extraction)
4. Create Phase 3 issue for parallel development

[bug-ops]

Resolution

After reviewing the actual implementation and agent behavior, this issue is resolved by the existing rust-agent-handoff skill — no additional code changes are needed.

How the skill addresses the MAST failure modes

The rust-agent-handoff skill (.zeph/skills/rust-agent-handoff/) is injected into every agent's system prompt via skills: include: [rust-agent-handoff] in agent definitions. It provides:

• Structured handoff protocol: agents write YAML files to .local/handoff/ with required fields (id, parent, agent, timestamp, status, context, output, next)
• Role-specific output schemas: references/{role}.md documents what each agent must include in their output — architects write decision_type, key_types, etc.; developers write files_modified, test_delta, etc.
• Context propagation: the skill instructs agents to read all parent handoff files before starting, ensuring context chains are preserved
• Verification gate: agents include acceptance_criteria coverage in their output, which the next agent can verify

The LLM model follows these instructions reliably — no compile-time struct enforcement or runtime schema validation is required. The skill IS the contract.

What was considered and rejected

Several implementation approaches were explored (typed HandoffContext enum, SchemaRegistry with YAML validation files, serde_norway parsing at dispatch time) but all introduced complexity without meaningful benefit: the model already produces well-structured handoff output when the skill is loaded.

Closing as resolved

Closing this issue. The 36.9% coordination failure rate identified by MAST is addressed through the skill-based protocol already in place. Follow-up: if empirical evidence from live sessions shows specific recurring handoff failures, targeted hardening can be filed as a new issue.