chore(examples): reproduction scripts for Gemini 3 thoughtSignature issues#15548
chore(examples): reproduction scripts for Gemini 3 thoughtSignature issues#15548gr2m wants to merge 3 commits into
Conversation
Adds examples/ai-functions/src/stream-text/gateway/issue-14196.ts, a Gateway-routed end-to-end reproduction for issue #14196 (Vertex rejecting Google AI Studio thoughtSignatures on mid-conversation failover). Run with AI_GATEWAY_API_KEY set: pnpm tsx src/stream-text/gateway/issue-14196.ts Three turns through the Gateway: Turn 1 — only:['google'] : captures signatures from Google AI Studio Turn 2 — only:['vertex'] : replays the conversation to Vertex (the failover path described in the issue) Turn 3 — order:['google','vertex'] : reporter's exact configuration Today all three turns return 200. Vertex accepts the Google-AI-Studio signatures (the bug as reported does not reproduce against gemini-3-flash-preview on this branch). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds examples/ai-functions/src/stream-text/gateway/issue-10344-zod-validation.ts to replicate hsheth2's post-close claim on issue #10344: "this is particularly happening when the tool call fails due to zod validation. When the tool calls succeed it does seem to pass the thought_signature properly." The script runs a sequential treasure-hunt chain on `google/gemini-3-flash` via the AI Gateway with `only: ['google']` and forces `execute()` to throw on the 3rd call. Today the SDK preserves the `thoughtSignature` on both the `tool-call` and the synthetic `tool-error` content parts; Gemini accepts the next request, the model retries the failing tool call, and the chain completes without a `thought_signature` 400 (8 steps total). The claim does not reproduce on this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Added a second reproduction script,
The script runs a sequential treasure-hunt chain on Result on The step where the tool fails looks like this — both the The next Gemini request is accepted (no Other post-close claims on #10344 not directly testable here:
|
Adds examples/ai-functions/src/stream-text/gateway/issue-10344-client-side-tools.ts, a deterministic reproduction of @pheuter's post-close note on issue #10344: "Gemini 3 models require a thoughtSignature on every functionCall part when replaying conversation history. The AI SDK normally preserves this through callProviderMetadata, but it's possible in certain edge cases for clients not to store/send it back. In our codebase, it can happen with client-side tool execution." The script defines a tool without `execute` (so the SDK does not auto-run it), captures the assistant tool-call + thoughtSignature from Turn 1, then runs two Turn 2 variants: - Turn 2A: rebuilds the history with providerOptions.google.thoughtSignature intact -> 200, model produces a response. - Turn 2B: rebuilds the history WITHOUT providerOptions on the tool-call (the "buggy client" path) -> Gemini 400: "Function call is missing a thought_signature in functionCall parts." So the SDK preserves the signature on the wire; the failure mode is application code that drops providerMetadata between turns when it serializes/deserializes assistant tool-call messages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Added a third reproduction script — Targets @pheuter's post-close note on #10344:
Setup: tool defined without
Sample output (trimmed): Interpretation: the SDK preserves the signature on the wire and Gemini validates it. The reported This narrows the issue substantially. Common surfaces where this happens:
For surfaces 1 and 2, the fix is on the application side: preserve Whether the SDK should do anything to mitigate this (e.g., emit a warning when a |
|
Filed follow-up: #15550 — proposes a Gemini-3-only runtime warning when an outbound request contains |
… replays (#15560) ## Background Gemini 3 rejects replayed assistant `functionCall` parts without `thoughtSignature` with HTTP 400. This happens when app/client code persists or rebuilds messages and drops `providerOptions`. ## Summary - For Gemini 3 only (`/^gemini-3[.-]/`), inject Google's documented `skip_thought_signature_validator` sentinel when a replayed tool call has no signature. - Warn once per request with affected tool names. - Adapted to `main` V4 Google provider files and existing `google` / `googleVertex` / `vertex` namespace lookup. ## Manual Verification Live replay against `google('gemini-3-flash-preview')`: missing-signature replay changed from HTTP 400 to HTTP 200 with warning. ## Checklist - [ ] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Related Issues Closes #15550. Refs #10344, #11413, #14196, #15548, #13060. Co-authored-by: Cursor <cursoragent@cursor.com>
… tool-call replays (#15553) ## Background Gemini 3 rejects replayed assistant `functionCall` parts without `thoughtSignature` with HTTP 400. This happens when app/client code persists or rebuilds messages and drops `providerOptions`. ## Summary - For Gemini 3 only (`/^gemini-3[.-]/`), inject Google's documented `skip_thought_signature_validator` sentinel when a replayed tool call has no signature. - Warn once per request with affected tool names. - Centralize signature namespace lookup and honor `google`, `googleVertex`, and `vertex` fallbacks. ## Manual Verification Live replay against `google('gemini-3-flash-preview')`: missing-signature replay changed from HTTP 400 to HTTP 200 with warning. ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Related Issues Closes #15550. Refs #10344, #11413, #14196, #15548, #13060. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Adds three reproduction scripts under
examples/ai-functions/src/stream-text/gateway/covering the activethought_signatureissues againstgoogle/gemini-3-flashonrelease-v6.0. Each script targets one specific claim from the issue tracker; together they map out where the bug is and is not.ai@6.0.190/@ai-sdk/gateway@3.0.119/@ai-sdk/google@3.0.79/@ai-sdk/google-vertex@4.0.135issue-14196.tsthoughtSignatureon Gateway failover (order: ['google','vertex']).only: ['google']) → 200 with 10 google-namespaced signatures. Turn 2 (only: ['vertex'], the failover sim) → 200; Vertex accepts the signatures. Turn 3 (reporter's exactorder: ['google','vertex']) → 200 on Google, no failover triggered.issue-10344-zod-validation.tsexecute()throw on the 3rd call. SDK preservesproviderMetadata.google.thoughtSignatureon both thetool-calland the synthetictool-errorcontent parts. Gemini accepts the next request, model retries, chain completes at step 7.issue-10344-client-side-tools.tsproviderMetadatabetween turns.Function call is missing a thought_signature in functionCall parts.What this tells us
thoughtSignaturecorrectly on the wire — across direct provider calls, Gateway routing, mid-chain tool failures, and the Vertex/Google namespace boundary (PR fix(google): resolve thoughtSignature from vertex namespace on gateway failover #13060).tool-callparts are serialized/persisted/re-hydrated without round-trippingproviderOptions(custom DB schemas,useChatserver routes that rebuild messages from the client payload, synthetic tool-call injection). When that happens, the next Gemini 3 request fails with HTTP 400.@ai-sdk/googlethat fires when an outbound Gemini 3 request containsfunctionCallparts with nothoughtSignatureunder any recognised namespace — making the cause visible before the 400 hits.How to run
Set
AI_GATEWAY_API_KEYto a key whose team allows both thegoogleandvertexproviders, then fromexamples/ai-functions/:Each script ends with a clear ✓/✗ summary. The third one is also useful as a smoke check — if
Turn 2Bever starts returning 200, Google has relaxed signature validation and #10344-style failures will quietly disappear; ifTurn 2Aever starts failing, the SDK is dropping signatures somewhere it shouldn't.Refs: #10344 #11413 #14196 #15550, related fix in #13060.
Test plan
AI_GATEWAY_API_KEYwhose team allowsgoogleandvertexprovidersissue-14196.ts: three Gateway turns all return 200 todayissue-10344-zod-validation.ts: chain completes despite mid-chain tool failure; signatures preserved on the error stepissue-10344-client-side-tools.ts: Turn 2A returns 200, Turn 2B returns 400 with the expected error wording — deterministicexamples/ai-functions