Recording a Haystack pipeline and replaying it with the provider unreachable

[xizhuomengcontin]

I maintain OrcaReplay (Apache-2.0), which records an agent or pipeline run from outside the process and can serve the recording back so the run happens again with no provider called. I checked it against Haystack tonight. Short version: it captures with no changes to your pipeline at all, and I want to correct something I had written down about Haystack a few days ago.

The correction first

My own notes said Haystack failed to capture: the client received the proxy URL but the request never arrived and the call timed out. I did not publish that anywhere, because I could not explain it and would not claim a framework was unsupported on one bad run.

Tonight I retested on the same version (haystack-ai 3.1.1, Python 3.12) in two shapes — the generator on its own and a real Pipeline — and both captured cleanly. I cannot reproduce the earlier failure, so the honest conclusion is that it was environmental on my side (most likely a system proxy variable in that shell), not a Haystack or OrcaReplay defect. Saying so here rather than letting a wrong note sit in my own docs.

Why it works with zero changes

OpenAIChatGenerator takes api_base_url defaulting to None and passes it straight through as the client's base_url. When that is None, the OpenAI SDK falls back to OPENAI_BASE_URL from the environment — which is exactly the variable orca record sets for the child process it launches, and only for that process.

So nothing in the pipeline changes:

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat.openai import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

pipe = Pipeline()
pipe.add_component("prompt", ChatPromptBuilder(
    template=[ChatMessage.from_user("{{question}}")], required_variables=["question"]))
pipe.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))
pipe.connect("prompt.prompt", "llm.messages")
pipe.run({"prompt": {"question": "Say hello in three words."}})

$ orca record generic-openai -- python pipeline.py
haystack pipeline replied: the stub answered
info recorded run=run_5381c38c5274 events=6 exit=0

Then, with the origin process killed:

$ orca replay last
info replaying exchanges=1 egress=blocked
haystack replied: the stub answered
info replay.done reused=1/1 exact=1 divergences=0 unmatched=0 exit=0

exact=1 means the replayed request matched the recording byte for byte — prompt rendering, message assembly and parameters all reproduced — with nothing upstream to talk to. Your components, your connections and your own code run for real; only the model's answer comes from the trace.

Where this is useful in a Haystack project

• A pipeline that produced a bad answer becomes an artifact a colleague can re-run without your API key and without spending anything — a bug report you can hand over rather than describe.
• The same recording is a regression test that costs nothing in CI, which is otherwise the awkward part of testing anything with a generator in it: real calls cost money, need a key, and do not answer the same way twice.
• Changing a ChatPromptBuilder template and replaying shows you the diff as a reported divergence, with its size, rather than a silently different run.

Two limits, stated rather than buried

• egress=blocked means model-provider egress, not network isolation. Replay stops the model call, but your components still run for real — a retriever still hits your document store, a custom component still makes its own calls. It is not a sandbox.
• A matching replay is not a determinism result. The model is not re-asked; its recorded answers are served back. Whether a fresh run would produce the same thing is a different question that replay cannot answer.

One version note for anyone trying this: OpenAIGenerator (the non-chat one) is gone in 3.1.1 — haystack.components.generators.chat.openai.OpenAIChatGenerator is the path that exists.

Tested with a deterministic local origin standing in for a provider, so the "same answer" above is a real byte comparison rather than a coincidence.