Skip to content

pi-sdk-azure: Azure endpoint is treated as an invalid URL in SDK target #931

@christso

Description

@christso

Summary

The pi-sdk-azure target fails against a real Azure-backed eval on main before scoring begins. The SDK path treats the configured Azure endpoint as an invalid URL.

Reproduction

From the AgentV repo root, with AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, and AZURE_DEPLOYMENT_NAME set in .env and GRADER_TARGET=azure:

GRADER_TARGET=azure bun apps/cli/src/cli.ts eval run \
  examples/features/deterministic-evaluators/evals/dataset.eval.yaml \
  --test-id equals-exact \
  --target pi-sdk-azure

Expected

The eval should complete and score the exact-match test against the Azure-backed pi SDK target.

Actual

The run fails with an execution error before scoring:

pi-coding-agent SDK error (provider: openai, model: gpt-5.4-mini): "leos-m6pmw8kz-eastus2/responses" cannot be parsed as a URL.

Notes

  • pi-azure does work in the same environment with a real Azure-backed eval.
  • This appears to be specific to the pi-coding-agent SDK Azure path, not the CLI path.
  • The error was reproduced on main on 2026-04-04.

Environment

  • Repo: EntityProcess/agentv
  • Branch: main
  • Target: pi-sdk-azure
  • Eval: examples/features/deterministic-evaluators/evals/dataset.eval.yaml
  • Test id: equals-exact

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions