Byte-exact replay of DSPy LM traffic — re-running a program with the model server switched off

[xizhuomengcontin]

Verified against DSPy 3.3.1 today: LM traffic from a DSPy program can be recorded from outside the process and served back later, so the program runs again with no model server present.

Disclosure: I work on the tool (OrcaReplay, Apache-2.0, no paid tier).

The run

$ orca record generic-openai -- python dspy_agent.py
dspy said: {'text': 'DSPY', 'reasoning_content': …}
info recorded run=run_0e2a338d33ce events=6 exit=0

$ taskkill /F /IM ollama.exe          # the model server is gone
$ orca replay last
dspy said: {'text': 'DSPY', 'reasoning_content': …}
info replay.done reused=1/1 exact=1 divergences=0 unmatched=0 exit=0

Nothing is installed into DSPy — no callback, no custom LM subclass. The origin is read from the environment, LiteLLM underneath does what it always does.

How this differs from cache=True

DSPy already caches, so the obvious question is why bother. Three differences that matter to me:

• The cache is keyed on the request; this stores the verbatim response bytes, so a replay exercises LiteLLM's and DSPy's own parsing rather than returning a deserialised object. Parsing bugs stay visible.
• It is a file in the project, so a colleague or CI can replay a run you recorded. A local cache cannot travel.
• It survives changes to your program. You can edit a signature or a module and re-run against the same recorded responses to see what your change did — the cache would miss on the new prompt, this either serves the recorded exchange or tells you plainly that the request diverged, with a character count.

The last one is the reason I find it useful with optimizers: a compile run makes many LM calls, and being able to replay one for free while you change the surrounding code is worth the disk.

What I actually tested, precisely

A raw lm(messages=[…]) call, not dspy.Predict. The local model I used (a 0.8B) cannot follow DSPy's structured output format, so Predict fails at parsing — that would have been testing the model, not the capture. The request and response themselves are captured and replayed identically either way, but I would rather state the boundary than let you assume I ran a full program.

Scope

egress=blocked on replay means model-provider egress. The rest of your program still runs for real — it is not a sandbox. And this sees the HTTP to the model, not DSPy's internals; it is not a substitute for MLflow-style tracing.

Happy to be told there is a neater way to do this with DSPy's own machinery.