Replayable Agent Mission Trace & Failure Attribution

[SumanMadipeddi]

Replayable Agent Mission Trace & Failure Attribution

I've been exploring a mission-level trace for the Go2 agentic pipeline that makes an execution reconstructable across:

user input → agent decision → tool selection → MCP execution → background activity → physical outcome

The main goal is to make it possible to answer: what happened during a mission, and where did it fail?

Current gap

Most of the underlying information already exists in DimOS, but across different layers:

• McpClient has agent/conversation state
• MCP / SkillResult has tool execution outcomes
• ToolStream carries background progress
• memory2 stores timestamped sensor/replay data
• runtime logs capture process-level behavior

There isn't currently one mission-correlated execution record tying those together.

Proposed architecture

Rather than introducing another observability stack, I built the trace around existing DimOS abstractions.

                       EXISTING EXECUTION

User → McpClient → MCP → McpServer → @skill → robot/replay/sim
          │                │             │
          │                │             │
          ▼                ▼             ▼
    client events     server events   ToolStream
          │                │          progress
          └────────────┬───┴──────────────┘
                       ▼
              MissionTraceRecorder
                       ▼
                    memory2
                       ▼
              mission_events stream
                       ▼
          <run-dir>/mission_trace.db
                       ▼
          deterministic attribution

Correlation follows:

run_id → mission_id → turn_id → invocation_id

where the existing DIMOS_RUN_ID and MCP progressToken are reused rather than introducing redundant identifiers.

A key semantic distinction is:

tool_completed = invocation returned
                 ≠
physical task verified

That becomes especially important for navigation/background skills.

What is working

The current prototype includes:

• client-side mission/agent events
• server-side tool execution events
• MCP correlation propagation
• ToolStream/background correlation
• correct mission boundaries for external input vs internal progress
• memory2 persistence through a mission_events stream
• deterministic failure attribution
• bounded/sanitized event payloads
• trace recording during replay

I've validated it with:

• unit tests
• mocked MCP + deterministic model tests
• ruff / mypy / blueprint registry checks
• Go2 replay end-to-end
• Go2 MuJoCo simulation end-to-end

In simulation I ran movement missions such as:

walk forward 1 meter
walk forward 1 meter with 30 degrees

and traced the model-selected relative_move invocation, arguments, server execution, duration, and result through the same mission correlation chain.

Tracing is currently opt-in through unitree-go2-agentic-trace; the existing unitree-go2-agentic execution path remains unchanged.

Next layer: physical verification

The main gap I want to tackle next is distinguishing:

"the skill returned successfully"

from

"the robot actually achieved the requested physical result."

My current direction is a non-invasive MissionVerifier using signals already present in the Go2 stack:

goal_reached
odom / TF
navigation state
perception evidence

so a physical mission can eventually produce:

physical_task_started
physical_task_completed
task_verification
mission_completed / mission_failed

with measured evidence rather than inferring success from an RPC response.

After that, I think the same trace can support:

1. sensor ↔ mission-event alignment through memory2
2. a small read-only mission timeline/debugging view
3. cross-run evals/regression testing across models, replay, simulation, and eventually hardware
4. standardizing long-running skills around stronger SkillResult / ToolStream terminal semantics

I've included the architecture diagram below, and the longer implementation/design notes are here:

Detailed design: https://docs.google.com/document/d/1VRBOK9BiP_m8GFj6sHeBcMx143DvK1CJCc81GuuM7ec/edit?tab=t.0#heading=h.u8psuqxrfybl

Prototype branch: https://github.com/SumanMadipeddi/dimos/tree/feat/agent-mission-trace

I'd especially appreciate feedback on:

• whether mission tracing should remain an opt-in composition or become a configurable capability on the existing agentic blueprint;
• whether MissionVerifier is the right next boundary for physical-task verification;
• which Go2 workflow would be the most useful first end-to-end verification target.