Skip to content

fix(core): drop api_format from Azure targets; surface pi-ai errors#1214

Merged
christso merged 2 commits intomainfrom
fix/azure-pi-ai-regression
May 4, 2026
Merged

fix(core): drop api_format from Azure targets; surface pi-ai errors#1214
christso merged 2 commits intomainfrom
fix/azure-pi-ai-regression

Conversation

@christso
Copy link
Copy Markdown
Collaborator

@christso christso commented May 4, 2026

Summary

Two regressions from the Vercel AI SDK → pi-ai migration (#1206) broke Azure evals end-to-end whenever a target had api_format: chat (or unset, on a config still defaulting to it):

  1. API version mismatch. The migration kept the api_format field but pi-ai's azure-openai-responses provider always hits /openai/v1/responses. With chat-style defaults still in place, AgentV sent ?api-version=2024-12-01-preview to the Responses path → Azure rejected with 400 "API version not supported" on every call.
  2. Silent error swallowing. invokePiAi never inspected pi-ai's stopReason. When pi-ai surfaced the 400 as { stopReason: 'error', errorMessage: '...', content: [] }, the adapter happily returned an empty assistant message → graders reported misleading "Unexpected EOF" JSON parse failures, completely hiding the underlying HTTP error.

Fix

  • Remove api_format from Azure targets entirely. Pi-ai's adapter only exposes the Responses path, so a chat/responses switch on this provider has no effect. Default Azure api version to v1 (matching /openai/v1/responses). Reject api_format on Azure targets at both validation and resolution time with a migration message pointing at provider: openai for chat-completions-only deployments. api_format remains supported on provider: openai.
  • Throw from invokePiAi when pi-ai returns stopReason: 'error'. The thrown message includes the parsed HTTP status so isRetryableError can decide retry behavior correctly.
  • Drop api_format from the repo's .agentv/targets.yaml and update the Azure provider docs with the migration path.

There's also a small follow-up commit doing pure biome format --write on apps/cli/src/self-update.ts and apps/cli/test/self-update.test.ts — pre-existing lint failures from #1213 that were blocking the pre-push hook.

Test plan

  • All 1752 core unit tests pass
  • All 502 CLI unit tests pass
  • Validator and resolver tests updated; both surface the migration error for api_format on Azure
  • Red UAT (main): agentv eval examples/features/basic/evals/dataset.eval.yaml --target default with AZURE_OPENAI_API_FORMAT=chat (or unset) → 7/7 tests fail; agent + grader return empty content; surface error: LLM grader "llm-grader" failed after 3 attempts (... Unexpected EOF) — skipped. Captured in conversation; pi-ai's actual response is 400 API version not supported.
  • Green UAT (this branch): Same eval, same .env, with the updated .agentv/targets.yamlcode-review-javascript: 98% PASS in 6.3s.
  • Migration error UAT: agentv validate on a targets file containing api_format: chat for an azure target → emits [targets[0].api_format] The 'api_format' field is no longer supported on Azure targets. ... use 'provider: openai' with a deployment-scoped 'base_url' instead. agentv eval against the same targets file refuses to start with the same error.
  • Silent-error fix UAT: Force AZURE_OPENAI_API_VERSION=bogus-version → eval surfaces ERROR: pi-ai call failed: HTTP 400 API version not supported (clear) instead of "Unexpected EOF" (misleading).

🤖 Generated with Claude Code

christso and others added 2 commits May 4, 2026 07:38
Two regressions from the Vercel AI SDK → pi-ai migration (#1206) broke
Azure evals end-to-end whenever a target had api_format=chat (or unset
on a config still defaulting to it):

1. The migration kept the api_format field but pi-ai's azure-openai-
   responses provider always hits /openai/v1/responses. With chat-style
   defaults still in place, AgentV sent ?api-version=2024-12-01-preview
   to the Responses path, which Azure rejects with 400 "API version
   not supported." Every eval call failed.

2. invokePiAi never inspected pi-ai's stopReason. When pi-ai surfaced
   the 400 as { stopReason: 'error', errorMessage: '...', content: [] },
   the adapter happily returned an empty assistant message, which
   downstream graders then reported as "Unexpected EOF" JSON parse
   failures — completely hiding the underlying HTTP error.

Fix:
- Remove api_format from Azure targets entirely. Pi-ai's adapter only
  exposes the Responses path, so a chat/responses switch on this
  provider has no effect. Default Azure api version to v1 (matching the
  /openai/v1/responses path). Reject api_format on Azure targets at both
  validation and resolution time with a migration message pointing at
  `provider: openai` for chat-completions-only deployments. api_format
  remains supported on `provider: openai`.
- Throw from invokePiAi when pi-ai returns stopReason 'error' so failures
  reach the surface and withRetry can apply its status-based retry
  policy. The thrown message includes the parsed HTTP status so
  isRetryableError can decide correctly.

Also drops api_format from the repo's .agentv/targets.yaml and updates
the Azure provider docs to document the migration path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure whitespace fix from `biome format --write` on apps/cli/src/self-update.ts
and apps/cli/test/self-update.test.ts. Pre-existing issue from #1213; the
pre-push hook fails on these files independently of any other change, so
fixing them here unblocks the bug-fix branch's push.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@cloudflare-workers-and-pages
Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: b69d902
Status: ✅  Deploy successful!
Preview URL: https://8b96c870.agentv.pages.dev
Branch Preview URL: https://fix-azure-pi-ai-regression.agentv.pages.dev

View logs

@christso christso marked this pull request as ready for review May 4, 2026 10:07
@christso christso merged commit 46572a8 into main May 4, 2026
4 checks passed
@christso christso deleted the fix/azure-pi-ai-regression branch May 4, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant