Replaying a LightRAG run offline: exact with a local embedder, and exactly where it stops with openai_embed

[xizhuomengcontin]

I record agent and pipeline runs at the HTTP boundary to the model provider — from outside the process — and serve the recording back so the run happens again with no provider called. I spent tonight finding out how much of a LightRAG run that actually covers, because for a RAG pipeline the answer is not simply "all of it". Posting both halves: the one that works exactly, and the one that does not.

Half one: with a local embedding function, the replay is byte-exact

LightRAG 1.5.7, gpt_4o_mini_complete for the LLM (which reaches the OpenAI SDK with no base_url, so it picks up OPENAI_BASE_URL — the variable the recorder sets for the child process only), and a deterministic local embedder so nothing but the LLM goes over HTTP:

rag = LightRAG(
    working_dir=wd,
    llm_model_func=gpt_4o_mini_complete,
    embedding_func=EmbeddingFunc(embedding_dim=32, func=local_embed),  # pure-Python, no HTTP
)
await rag.ainsert("Order 4417 was shipped on Tuesday from the Rotterdam warehouse.")
ans = await rag.aquery("Where did order 4417 ship from?", param=QueryParam(mode="naive"))

Recorded a full ingest + query, then killed the model server and re-ran it against the recording:

info replay.done reused=3/3 exact=3 divergences=0 unmatched=0 exit=0

All three LLM exchanges — entity extraction during ingest, and the query — reproduced byte for byte with nothing upstream to reach. No code change, no key, nothing installed into LightRAG.

That makes an ingest+query pass into an artifact: a colleague can re-run your exact pipeline behaviour without your key and without spending anything, and the same recording pins the prompts in CI so a change to entity extraction or the query template shows up as a reported divergence rather than a silently different graph.

Half two: openai_embed traffic is not captured — measured, not assumed

Swapping the embedder for openai_embed(model="text-embedding-3-small") and recording again:

INFO:httpx2:HTTP Request: POST http://127.0.0.1:53898/v1/embeddings "HTTP/1.1 401 Unauthorized"
ERROR: Aborting pipeline batch due to storage flush error:
       NanoVectorDBStorage[chunks] index flush failed: Error code: 401 ...

The request did reach my proxy (that is its port), but the proxy does not claim /v1/embeddings as a model exchange — it has dialects for /v1/chat/completions, /v1/messages and the two Responses paths, and nothing else. So embedding traffic is passed straight through to the real provider: it is not redirected to a stand-in, and it is not written into the recording. Worse, at record time there is no warning — the run can exit 0 with a recording that looks complete.

On replay, that gap is at least loud rather than silent. The call is refused rather than leaked:

/v1/embeddings was captured as opaque network traffic, not as a model exchange,
so it cannot be replayed.

and the verdict line carries unmatched=N.

What that means concretely for LightRAG

• Embed at ingest, query with chat only → replays fully. Build the index once, keep it, and the query path is pure LLM traffic.
• Embed at query time over HTTP (the default with openai_embed, since the query itself is embedded) → cannot be fully replayed today. You would get the LLM half back and an explicit refusal on the embedding call.
• Local or in-process embedder → replays fully, as above.

This is a gap in my tool, not in LightRAG, and I am saying so here rather than writing a post that implies RAG is covered when a third of the traffic in a typical setup is not. I am adding an embeddings dialect; until it exists, the boundary above is the honest one.

Other limits worth stating

• Blocked model egress is not network isolation. Replay re-executes everything the run did apart from the model calls — your storage writes happen again, and any other HTTP a custom function makes goes out for real.
• A matching replay is not a determinism result. The model is not re-asked; its recorded answers are served back. It answers "do these outputs follow from these model responses", not "would the model respond this way again".
• LightRAG caches LLM responses on disk. A replay started against an already-populated working_dir may legitimately make fewer calls than the recording — that shows as a lower reused count and is not a failure.

Tested against a deterministic local origin standing in for a provider, so "byte for byte" above is a real comparison rather than a model happening to repeat itself. Tool is OrcaReplay (Apache-2.0, npm, Node 20+); I maintain it, so weigh accordingly. Corrections welcome — particularly if there is a LightRAG-native way to pin a query pass that I have missed.