Repository navigation
Recording traffic through the Gateway and replaying it with both the Gateway and the model gone #1801
xizhuomengcontin
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Tried something today and it worked, so sharing it: putting a recording proxy in front of the Gateway, then replaying the run with the Gateway and the model provider both switched off.
Disclosure: I work on the recording tool (OrcaReplay, Apache-2.0). It is not a gateway and does not compete with one — it does not route, load-balance or fall back. It records.
The chain
The app is unchanged; it just points
OPENAI_BASE_URLat the recorder, and thex-portkey-provider/x-portkey-custom-hostheaders ride along with the request as usual.Then I killed both processes — the Gateway and Ollama:
(I checked this twice, because the first time I only managed to kill Ollama and I did not want to claim more than I had actually shown.)
Why this might interest people here
Everything the Gateway does — routing, retries, guardrails, caching — is configuration whose effect you usually only see by sending a real request. Recording in front of it means you can:
It sits in front of, not instead of. If you record behind the Gateway instead you capture provider traffic and lose the Gateway's contribution, which is usually the part in question.
Scope
egress=blockedon replay means model-provider egress. Everything else the program does still runs for real; it is not a sandbox.Happy to be told there is a better place to tap this, or that the Gateway already exposes what I am reconstructing.
All reactions