Repository navigation
Recording a prompty flow and replaying it offline — and one misleading error message #4222
Xizhuo (xizhuomengcontin)
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I maintain OrcaReplay (Apache-2.0), which records an agent run from outside the process and can serve the recording back so the run happens again with no provider called. I checked it against promptflow this evening. Two things came out of it that seem worth sharing here: the good news that prompty flows capture cleanly with no changes at all, and one error message that sent me the wrong way for a few minutes.
Prompty flows are capturable with zero changes
This is not true of every framework — plenty of them pin the provider origin in their own config, which puts it out of reach of anything working at the process boundary. promptflow does the right thing: a prompty
configurationwithtype: openaiand no explicitbase_urlgoes throughOpenAIConnection.from_env(), which honoursOPENAI_BASE_URL.So there is nothing to change in the flow. The prompty file stays exactly as you wrote it:
Record it, then replay with the provider gone:
exact=1means the replayed request matched the recorded one byte for byte — template rendering, message assembly and parameters all reproduced — and the origin was not running. promptflow's own code path executes for real; only the model's answer comes from the recording.Why that is useful here specifically: a prompty flow that misbehaves becomes a fixture. Someone else can reproduce it without your key and without spending anything, and the same recording runs in CI as a regression test for template or parameter changes.
Tested against
promptflow-core1.18.5 /openai3.13.0 on Python 3.12, with a deterministic local origin standing in for a provider.The error message that cost me a few minutes
My first prompty used
type: openai_model, which I had picked up from a model-configuration example. It fails — fair enough, the accepted values areOpenAI/AzureOpenAI— but the message is:My flow has
api: chatand no embeddings anywhere in it. The string is hardcoded inconvert_model_configuration_to_connection(promptflow/core/_prompty_utils.py, around line 130) and always says "for embedding api" regardless of which API the caller is on, because that helper is shared:I went looking for an embedding step in my own flow before re-reading it. Dropping the two words, or interpolating the actual
api, would make it point at the real problem. The second sentence is already doing the useful work.Happy to open an issue or a one-line PR for that if it is not already known — I did not want to file noise if this is deliberate.
Two limits, stated rather than buried
egress=blockedmeans model-provider egress, not network isolation. Replay stops the model call, but the flow's own code still runs for real — a tool or script that reaches the network reaches it again. It is not a sandbox.All reactions