Summary
The pi-sdk-azure target fails against a real Azure-backed eval on main before scoring begins. The SDK path treats the configured Azure endpoint as an invalid URL.
Reproduction
From the AgentV repo root, with AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, and AZURE_DEPLOYMENT_NAME set in .env and GRADER_TARGET=azure:
GRADER_TARGET=azure bun apps/cli/src/cli.ts eval run \
examples/features/deterministic-evaluators/evals/dataset.eval.yaml \
--test-id equals-exact \
--target pi-sdk-azure
Expected
The eval should complete and score the exact-match test against the Azure-backed pi SDK target.
Actual
The run fails with an execution error before scoring:
pi-coding-agent SDK error (provider: openai, model: gpt-5.4-mini): "leos-m6pmw8kz-eastus2/responses" cannot be parsed as a URL.
Notes
pi-azure does work in the same environment with a real Azure-backed eval.
- This appears to be specific to the
pi-coding-agent SDK Azure path, not the CLI path.
- The error was reproduced on
main on 2026-04-04.
Environment
- Repo:
EntityProcess/agentv
- Branch:
main
- Target:
pi-sdk-azure
- Eval:
examples/features/deterministic-evaluators/evals/dataset.eval.yaml
- Test id:
equals-exact
Summary
The
pi-sdk-azuretarget fails against a real Azure-backed eval onmainbefore scoring begins. The SDK path treats the configured Azure endpoint as an invalid URL.Reproduction
From the AgentV repo root, with
AZURE_OPENAI_ENDPOINT,AZURE_OPENAI_API_KEY, andAZURE_DEPLOYMENT_NAMEset in.envandGRADER_TARGET=azure:GRADER_TARGET=azure bun apps/cli/src/cli.ts eval run \ examples/features/deterministic-evaluators/evals/dataset.eval.yaml \ --test-id equals-exact \ --target pi-sdk-azureExpected
The eval should complete and score the exact-match test against the Azure-backed pi SDK target.
Actual
The run fails with an execution error before scoring:
Notes
pi-azuredoes work in the same environment with a real Azure-backed eval.pi-coding-agentSDK Azure path, not the CLI path.mainon 2026-04-04.Environment
EntityProcess/agentvmainpi-sdk-azureexamples/features/deterministic-evaluators/evals/dataset.eval.yamlequals-exact