Skip to content

fix(ai-sdk): keep output validation enabled for tool procedures - #1791

Merged
dinwwwh merged 1 commit into
middleapi:mainfrom
dinwwwh:fix/ai-sdk-output-validation
Aug 5, 2026
Merged

fix(ai-sdk): keep output validation enabled for tool procedures#1791
dinwwwh merged 1 commit into
middleapi:mainfrom
dinwwwh:fix/ai-sdk-output-validation

Conversation

@dinwwwh

@dinwwwh dinwwwh commented Aug 5, 2026

Copy link
Copy Markdown
Member

createToolFactory disabled both input and output validation on the assumption that the AI SDK re-validates against the tool schemas. That is only true for input: the AI SDK treats outputSchema as type metadata and never validates the value execute returns against it (confirmed in vercel/ai#10222, docs corrected in vercel/ai#11016). Procedures run as tools therefore skipped their .output() schemas entirely, including transforms and defaults.

Fixes

  • Output validation stays enabled; only input validation remains disabled, since that half is genuinely redundant.
  • A handler returning invalid output now rejects with an output validation error instead of passing through silently.
  • Streamed asyncIteratorObject outputs validate every yielded event, and a handler returning a non-iterator now errors instead of being yielded once as the final result (the old fallback branch became unreachable and is removed).

Testing

  • A new generateText test with a mock model proves the AI SDK returns schema-violating execute output untouched, so it will start failing if a future ai release adds output validation and makes ours redundant.
  • Remaining tests cover input validation being skipped, invalid output rejection, per-event stream validation, and non-iterator rejection.

@vercel

vercel Bot commented Aug 5, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
orpc Ready Ready Preview Aug 5, 2026 7:45am

@pkg-pr-new

pkg-pr-new Bot commented Aug 5, 2026

Copy link
Copy Markdown
More templates

@orpc/ai-sdk

npm i https://pkg.pr.new/@orpc/ai-sdk@1791

@orpc/arktype

npm i https://pkg.pr.new/@orpc/arktype@1791

@orpc/bun

npm i https://pkg.pr.new/@orpc/bun@1791

@orpc/client

npm i https://pkg.pr.new/@orpc/client@1791

@orpc/cloudflare

npm i https://pkg.pr.new/@orpc/cloudflare@1791

@orpc/contract

npm i https://pkg.pr.new/@orpc/contract@1791

@orpc/experimental-effect

npm i https://pkg.pr.new/@orpc/experimental-effect@1791

@orpc/evlog

npm i https://pkg.pr.new/@orpc/evlog@1791

@orpc/hibernation

npm i https://pkg.pr.new/@orpc/hibernation@1791

@orpc/json-schema

npm i https://pkg.pr.new/@orpc/json-schema@1791

@orpc/nest

npm i https://pkg.pr.new/@orpc/nest@1791

@orpc/next

npm i https://pkg.pr.new/@orpc/next@1791

@orpc/openapi

npm i https://pkg.pr.new/@orpc/openapi@1791

@orpc/opentelemetry

npm i https://pkg.pr.new/@orpc/opentelemetry@1791

@orpc/pinia-colada

npm i https://pkg.pr.new/@orpc/pinia-colada@1791

@orpc/pino

npm i https://pkg.pr.new/@orpc/pino@1791

@orpc/publisher

npm i https://pkg.pr.new/@orpc/publisher@1791

@orpc/ratelimit

npm i https://pkg.pr.new/@orpc/ratelimit@1791

@orpc/server

npm i https://pkg.pr.new/@orpc/server@1791

@orpc/shared

npm i https://pkg.pr.new/@orpc/shared@1791

@orpc/swr

npm i https://pkg.pr.new/@orpc/swr@1791

@orpc/tanstack-query

npm i https://pkg.pr.new/@orpc/tanstack-query@1791

@orpc/trpc

npm i https://pkg.pr.new/@orpc/trpc@1791

@orpc/valibot

npm i https://pkg.pr.new/@orpc/valibot@1791

@orpc/zod

npm i https://pkg.pr.new/@orpc/zod@1791

commit: a5b928a

@codecov

codecov Bot commented Aug 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 25 untouched benchmarks


Comparing dinwwwh:fix/ai-sdk-output-validation (a5b928a) with main (9f9d664)

Open in CodSpeed

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes: Reviewed the full diff of commit a5b928a (the repository's only commit for this PR) in packages/ai-sdk/src/tool.ts and packages/ai-sdk/src/tool.test.ts.

  • Output validation re-enabledcreateToolFactory now disables only disableInputValidation; disableOutputValidation stays on, so tool procedures validate .output() schemas (including transforms/defaults) that the AI SDK never validates.
  • Iterator execute simplified — replaced the isAsyncIteratorObject fallback with yield* await call(...), relying on the (now enabled) asyncIteratorObject validation to reject non-iterators and validate each yielded event.
  • New premise-guard test — a generateText + MockLanguageModelV4 test proves the AI SDK passes schema-violating execute output through untouched, and would fail if a future ai release adds output validation.
  • Reworked tests — cover input validation being skipped, invalid output rejection, per-event stream validation, and non-iterator rejection.

I traced the behavior through procedure-client.ts (validateOutput/validateInput) and the asyncIteratorObject schema in packages/contract/src/schema-built-in.ts: validate() returns { issues } for non-iterators (→ "Output validation failed") or a wrapped iterator that validates each event (→ "AsyncIteratorObject validation failed"). The new tests map onto exactly these code paths and each genuinely fails under the prior validation-disabled behavior. All 20 tests in tool.test.ts pass and lint is clean. The single caveat—the generateText test uses current AI SDK v4 shapes and is version-fragile test infra—is acknowledged and not a code concern.

This is a correct and well-tested fix that closes a real validation gap.

Pullfrog  | View workflow run | Using openrouter/~deepseek/deepseek-v4-flash-latest (free via Pullfrog for OSS) | 𝕏

@dinwwwh
dinwwwh merged commit f36f85e into middleapi:main Aug 5, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant