Agent debugger: step through an agent's tool calls and intermediate results

[jaylfc]

When an agent produces a bad answer, today's option is to stare at logs. A proper debugger would let the user step through the agent's execution — see the LLM call, the tool invocation, the intermediate state, the next LLM call — and pause at any step.

What needs to ship:

• Record every agent execution as a structured trace: LLM call → tool call → result → next LLM call
• Debugger UI panel in the agent chat window that shows the trace as a tree
• Click a node to see inputs/outputs, token counts, wall time
• "Replay from here" — fork the agent from an intermediate step with a modified input and see how the rest plays out differently
• "Re-run with different model" — swap the LLM backend at a specific step and re-execute downstream
• Exports a replay file for sharing bug reports

User impact: debugging agents today is "cross your fingers and re-run"; a real debugger would turn agent development from vibes-based to engineering.

[m13v]

the "replay from here" feature would be incredibly valuable. we built structured tool execution tracing for our agent and the single biggest productivity gain was being able to see exactly which tool call produced unexpected output and re-run from that point. one thing to add to the trace: capture the token count and latency per LLM call, because often the real bug is that the context window filled up and the model started dropping important information from earlier in the chain. for the "re-run with different model" feature, also consider letting users edit the tool result at a specific step and see how the downstream LLM calls change. sometimes the issue is not the model but a tool returning malformed data, and being able to inject a corrected result mid-trace is faster than fixing the tool and re-running everything.

[m13v]

our tool execution layer that captures structured traces of each tool call with inputs, outputs, and timing: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift

[jaylfc]

Thanks @m13v — "replay from here" is exactly the differentiator between a log viewer and a real debugger. Being able to mutate the state at any step and re-run from that point turns debugging from "guess and re-run the whole thing" into "surgical fix and verify."

Your ChatToolExecutor.swift trace structure is a good reference. For taOS we'd capture traces in the taOSmd zero-loss archive — every tool call already gets archived with inputs/outputs. The debugger UI would read these traces and render them as a step-through tree. The archive gives us "replay from here" for free since we have the full state at every step.

This ties into the taOSmd memory system nicely — the archive IS the execution trace, the debugger is just a UI over it.

[m13v]

one subtlety worth flagging: the archive gives you the data for replay, but "replay from here" with a mutated tool result means you need to re-execute downstream LLM calls, not just read them from the archive. that requires the prompt construction logic and model access at replay time, not just the stored inputs/outputs. the archive tells you what happened, but the executor is what lets you ask "what would have happened if this tool returned X instead." so the debugger is a UI over the archive for inspection, but it's a UI over the executor for replay. might be worth keeping those two modes (inspect vs. re-execute) explicit in the design so the archive format doesn't accidentally couple to execution concerns.

[jaylfc]

Spot on — the debugger is a UI over the archive for inspection, but "replay from here" requires a live executor with prompt construction + model access. Two distinct components:

1. Inspector (archive-only, read-only) — timeline view, step-through, state at each point. This is what the session catalog gives us now.

2. Executor (requires LLM) — "what if this tool returned X instead?" Re-constructs the prompt from that step's state, calls the model, runs the downstream chain. This is the expensive part and it's where the resource manager matters — you need a model available to re-execute.

The session catalog + crystal digests handle #1 well. For #2, the job queue can schedule re-execution jobs with the right model priority. The zero-loss archive ensures we have the exact inputs at every step, so prompt reconstruction is deterministic.

Good distinction to keep in mind as we build this.