fix(core): drop api_format from Azure targets; surface pi-ai errors#1214
Merged
fix(core): drop api_format from Azure targets; surface pi-ai errors#1214
Conversation
Two regressions from the Vercel AI SDK → pi-ai migration (#1206) broke Azure evals end-to-end whenever a target had api_format=chat (or unset on a config still defaulting to it): 1. The migration kept the api_format field but pi-ai's azure-openai- responses provider always hits /openai/v1/responses. With chat-style defaults still in place, AgentV sent ?api-version=2024-12-01-preview to the Responses path, which Azure rejects with 400 "API version not supported." Every eval call failed. 2. invokePiAi never inspected pi-ai's stopReason. When pi-ai surfaced the 400 as { stopReason: 'error', errorMessage: '...', content: [] }, the adapter happily returned an empty assistant message, which downstream graders then reported as "Unexpected EOF" JSON parse failures — completely hiding the underlying HTTP error. Fix: - Remove api_format from Azure targets entirely. Pi-ai's adapter only exposes the Responses path, so a chat/responses switch on this provider has no effect. Default Azure api version to v1 (matching the /openai/v1/responses path). Reject api_format on Azure targets at both validation and resolution time with a migration message pointing at `provider: openai` for chat-completions-only deployments. api_format remains supported on `provider: openai`. - Throw from invokePiAi when pi-ai returns stopReason 'error' so failures reach the surface and withRetry can apply its status-based retry policy. The thrown message includes the parsed HTTP status so isRetryableError can decide correctly. Also drops api_format from the repo's .agentv/targets.yaml and updates the Azure provider docs to document the migration path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure whitespace fix from `biome format --write` on apps/cli/src/self-update.ts and apps/cli/test/self-update.test.ts. Pre-existing issue from #1213; the pre-push hook fails on these files independently of any other change, so fixing them here unblocks the bug-fix branch's push. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploying agentv with
|
| Latest commit: |
b69d902
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://8b96c870.agentv.pages.dev |
| Branch Preview URL: | https://fix-azure-pi-ai-regression.agentv.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two regressions from the Vercel AI SDK → pi-ai migration (#1206) broke Azure evals end-to-end whenever a target had
api_format: chat(or unset, on a config still defaulting to it):api_formatfield but pi-ai'sazure-openai-responsesprovider always hits/openai/v1/responses. With chat-style defaults still in place, AgentV sent?api-version=2024-12-01-previewto the Responses path → Azure rejected with400 "API version not supported"on every call.invokePiAinever inspected pi-ai'sstopReason. When pi-ai surfaced the 400 as{ stopReason: 'error', errorMessage: '...', content: [] }, the adapter happily returned an empty assistant message → graders reported misleading "Unexpected EOF" JSON parse failures, completely hiding the underlying HTTP error.Fix
api_formatfrom Azure targets entirely. Pi-ai's adapter only exposes the Responses path, so a chat/responses switch on this provider has no effect. Default Azure api version tov1(matching/openai/v1/responses). Rejectapi_formaton Azure targets at both validation and resolution time with a migration message pointing atprovider: openaifor chat-completions-only deployments.api_formatremains supported onprovider: openai.invokePiAiwhen pi-ai returnsstopReason: 'error'. The thrown message includes the parsed HTTP status soisRetryableErrorcan decide retry behavior correctly.api_formatfrom the repo's.agentv/targets.yamland update the Azure provider docs with the migration path.There's also a small follow-up commit doing pure
biome format --writeonapps/cli/src/self-update.tsandapps/cli/test/self-update.test.ts— pre-existing lint failures from #1213 that were blocking the pre-push hook.Test plan
api_formaton Azureagentv eval examples/features/basic/evals/dataset.eval.yaml --target defaultwithAZURE_OPENAI_API_FORMAT=chat(or unset) → 7/7 tests fail; agent + grader return empty content; surface error:LLM grader "llm-grader" failed after 3 attempts (... Unexpected EOF) — skipped. Captured in conversation; pi-ai's actual response is400 API version not supported..env, with the updated.agentv/targets.yaml→code-review-javascript: 98% PASSin 6.3s.agentv validateon a targets file containingapi_format: chatfor an azure target → emits[targets[0].api_format] The 'api_format' field is no longer supported on Azure targets. ... use 'provider: openai' with a deployment-scoped 'base_url' instead.agentv evalagainst the same targets file refuses to start with the same error.AZURE_OPENAI_API_VERSION=bogus-version→ eval surfacesERROR: pi-ai call failed: HTTP 400 API version not supported(clear) instead of "Unexpected EOF" (misleading).🤖 Generated with Claude Code