Skip to content

chore(examples): reproduction scripts for Gemini 3 thoughtSignature issues#15548

Draft
gr2m wants to merge 3 commits into
release-v6.0from
gr2m/repro-14196
Draft

chore(examples): reproduction scripts for Gemini 3 thoughtSignature issues#15548
gr2m wants to merge 3 commits into
release-v6.0from
gr2m/repro-14196

Conversation

@gr2m
Copy link
Copy Markdown
Collaborator

@gr2m gr2m commented May 22, 2026

Summary

Adds three reproduction scripts under examples/ai-functions/src/stream-text/gateway/ covering the active thought_signature issues against google/gemini-3-flash on release-v6.0. Each script targets one specific claim from the issue tracker; together they map out where the bug is and is not.

File Issue / claim targeted Outcome on ai@6.0.190 / @ai-sdk/gateway@3.0.119 / @ai-sdk/google@3.0.79 / @ai-sdk/google-vertex@4.0.135
issue-14196.ts #14196 — Vertex rejects Google-AI-Studio thoughtSignature on Gateway failover (order: ['google','vertex']). Not reproduced. Turn 1 (only: ['google']) → 200 with 10 google-namespaced signatures. Turn 2 (only: ['vertex'], the failover sim) → 200; Vertex accepts the signatures. Turn 3 (reporter's exact order: ['google','vertex']) → 200 on Google, no failover triggered.
issue-10344-zod-validation.ts #10344 (post-close, hsheth2) — "tool call fails due to zod validation" drops the signature. Not reproduced. Forced execute() throw on the 3rd call. SDK preserves providerMetadata.google.thoughtSignature on both the tool-call and the synthetic tool-error content parts. Gemini accepts the next request, model retries, chain completes at step 7.
issue-10344-client-side-tools.ts #10344 (post-close, pheuter) — client-side tool execution path where the application drops providerMetadata between turns. Reproduced deterministically. Turn 2A (signature preserved) → 200. Turn 2B (signature stripped from the assistant tool-call) → HTTP 400 Function call is missing a thought_signature in functionCall parts.

What this tells us

How to run

Set AI_GATEWAY_API_KEY to a key whose team allows both the google and vertex providers, then from examples/ai-functions/:

pnpm tsx src/stream-text/gateway/issue-14196.ts
pnpm tsx src/stream-text/gateway/issue-10344-zod-validation.ts
pnpm tsx src/stream-text/gateway/issue-10344-client-side-tools.ts

Each script ends with a clear ✓/✗ summary. The third one is also useful as a smoke check — if Turn 2B ever starts returning 200, Google has relaxed signature validation and #10344-style failures will quietly disappear; if Turn 2A ever starts failing, the SDK is dropping signatures somewhere it shouldn't.

Refs: #10344 #11413 #14196 #15550, related fix in #13060.

Test plan

  • All three scripts run cleanly with a valid AI_GATEWAY_API_KEY whose team allows google and vertex providers
  • issue-14196.ts: three Gateway turns all return 200 today
  • issue-10344-zod-validation.ts: chain completes despite mid-chain tool failure; signatures preserved on the error step
  • issue-10344-client-side-tools.ts: Turn 2A returns 200, Turn 2B returns 400 with the expected error wording — deterministic
  • Type-checks under examples/ai-functions

Adds examples/ai-functions/src/stream-text/gateway/issue-14196.ts, a
Gateway-routed end-to-end reproduction for issue #14196 (Vertex rejecting
Google AI Studio thoughtSignatures on mid-conversation failover).

Run with AI_GATEWAY_API_KEY set:
  pnpm tsx src/stream-text/gateway/issue-14196.ts

Three turns through the Gateway:
  Turn 1 — only:['google'] : captures signatures from Google AI Studio
  Turn 2 — only:['vertex'] : replays the conversation to Vertex
                              (the failover path described in the issue)
  Turn 3 — order:['google','vertex'] : reporter's exact configuration

Today all three turns return 200. Vertex accepts the Google-AI-Studio
signatures (the bug as reported does not reproduce against
gemini-3-flash-preview on this branch).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds examples/ai-functions/src/stream-text/gateway/issue-10344-zod-validation.ts
to replicate hsheth2's post-close claim on issue #10344:

  "this is particularly happening when the tool call fails due to zod
  validation. When the tool calls succeed it does seem to pass the
  thought_signature properly."

The script runs a sequential treasure-hunt chain on `google/gemini-3-flash`
via the AI Gateway with `only: ['google']` and forces `execute()` to throw
on the 3rd call.

Today the SDK preserves the `thoughtSignature` on both the `tool-call`
and the synthetic `tool-error` content parts; Gemini accepts the next
request, the model retries the failing tool call, and the chain completes
without a `thought_signature` 400 (8 steps total). The claim does not
reproduce on this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@gr2m
Copy link
Copy Markdown
Collaborator Author

gr2m commented May 22, 2026

Added a second reproduction script, examples/ai-functions/src/stream-text/gateway/issue-10344-zod-validation.ts, targeting the post-close claim on #10344 by @hsheth2:

For me this is particularly happening when the tool call fails due to zod validation. When the tool calls succeed it does seem to pass the thought_signature properly.

The script runs a sequential treasure-hunt chain on google/gemini-3-flash via the Gateway (only: ['google']) and forces execute() to throw on the 3rd tool call.

Result on ai@6.0.190 / @ai-sdk/google@3.0.79 / @ai-sdk/gateway@3.0.119: claim does not reproduce.

The step where the tool fails looks like this — both the tool-call and the synthetic tool-error carry the same thoughtSignature:

[step 2] finishReason=tool-calls toolCalls=1 toolResults=0
  tool-call  clue(mailbox)  sigPrefix=Et4BCtsBAQw5
  tool-error clue(mailbox)  errorMsg="BOOM: simulated tool failure..." sigPrefix=Et4BCtsBAQw5

The next Gemini request is accepted (no thought_signature 400), the model retries the failing call with a fresh signature, and the chain completes at step 7. So the SDK's tool-error → tool-result conversion in packages/ai/src/generate-text/to-response-messages.ts preserves the signature correctly on failed calls today.

Other post-close claims on #10344 not directly testable here:

  • @pheuter (client-side tool execution edge case) — different code path; would need a separate script.
  • @shtefcs (multi-agent handoff) — partially covered by the Turn 1 → Turn 2 split in issue-14196.ts, which already works.
  • @benevbright — no repro details provided.

Adds examples/ai-functions/src/stream-text/gateway/issue-10344-client-side-tools.ts,
a deterministic reproduction of @pheuter's post-close note on issue
#10344:

  "Gemini 3 models require a thoughtSignature on every functionCall part
  when replaying conversation history. The AI SDK normally preserves this
  through callProviderMetadata, but it's possible in certain edge cases
  for clients not to store/send it back. In our codebase, it can happen
  with client-side tool execution."

The script defines a tool without `execute` (so the SDK does not auto-run
it), captures the assistant tool-call + thoughtSignature from Turn 1,
then runs two Turn 2 variants:

  - Turn 2A: rebuilds the history with providerOptions.google.thoughtSignature
             intact -> 200, model produces a response.
  - Turn 2B: rebuilds the history WITHOUT providerOptions on the tool-call
             (the "buggy client" path) -> Gemini 400:
             "Function call is missing a thought_signature in functionCall
             parts."

So the SDK preserves the signature on the wire; the failure mode is
application code that drops providerMetadata between turns when it
serializes/deserializes assistant tool-call messages.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@gr2m
Copy link
Copy Markdown
Collaborator Author

gr2m commented May 22, 2026

Added a third reproduction script — examples/ai-functions/src/stream-text/gateway/issue-10344-client-side-tools.ts — and this one reproduces the bug deterministically.

Targets @pheuter's post-close note on #10344:

Gemini 3 models require a thoughtSignature on every functionCall part when replaying conversation history. The AI SDK normally preserves this through callProviderMetadata, but it's possible in certain edge cases for clients not to store/send it back. In our codebase, it can happen with client-side tool execution.

Setup: tool defined without execute, so the SDK does not auto-run it (client-side execution path). Turn 1 captures the assistant tool-call + its providerOptions.google.thoughtSignature. Turn 2 has two variants:

Turn History shape Result on ai@6.0.190 + @ai-sdk/gateway@3.0.119 + google/gemini-3-flash
2A tool-call with providerOptions.google.thoughtSignature ✓ 200 — model produces a response
2B tool-call without providerOptions HTTP 400Function call is missing a thought_signature in functionCall parts. […] Additional data, function call \default_api:weather` , position 2.`

Sample output (trimmed):

[Turn 1] tool-call 7sokhw95 input={"location":"San Francisco"} sigPrefix=Eo0DCooDAQw51sc3... (len=536)
[client] executed tool externally, result: {"location":"San Francisco","temperature":72,"condition":"sunny"}

=== Turn 2A: well-behaved client — signature preserved on tool-call ===
✓ Turn 2A (signature preserved) succeeded.

=== Turn 2B: buggy client — providerMetadata stripped from tool-call ===
✗ Turn 2B (signature stripped) FAILED:
  statusCode: 400
  message: Function call is missing a thought_signature in functionCall parts. […]

--- Summary ---
  Turn 2A (well-behaved): succeeded ✓ (expected)
  Turn 2B (signature stripped): failed with 400 ✓ — bug reproduced (expected)

Interpretation: the SDK preserves the signature on the wire and Gemini validates it. The reported thought_signature 400s are an application-layer failure mode — application code that serializes/deserializes assistant tool-call messages without round-tripping providerOptions (or callProviderMetadata in older APIs) loses the signature and the next Gemini call fails.

This narrows the issue substantially. Common surfaces where this happens:

  • Persisting chat history to a database with a custom schema that drops providerMetadata
  • useChat round-trips where the server route reconstructs messages from the client payload without the providerMetadata field
  • Custom tool result injection (e.g. inserting a synthetic functionCall block to seed an agent state)

For surfaces 1 and 2, the fix is on the application side: preserve providerMetadata (or providerOptions) on every tool-call part across persistence and HTTP boundaries. For surface 3, Google documents "skip_thought_signature_validator" as a dummy sentinel (also noted by @pheuter upthread).

Whether the SDK should do anything to mitigate this (e.g., emit a warning when a tool-call is replayed without a signature for Gemini 3 models, or auto-inject skip_thought_signature_validator in known synthetic-injection paths) is a follow-up design question; not in scope for this PR.

@gr2m
Copy link
Copy Markdown
Collaborator Author

gr2m commented May 22, 2026

Filed follow-up: #15550 — proposes a Gemini-3-only runtime warning when an outbound request contains functionCall parts with no thoughtSignature under any recognised namespace. Warning only (no throw, no auto-inject); intent is to make the application-layer cause visible before the inevitable 400.

@gr2m gr2m changed the title chore(examples): add reproduction script for #14196 chore(examples): reproduction scripts for Gemini 3 thoughtSignature issues May 22, 2026
gr2m added a commit that referenced this pull request May 27, 2026
… replays (#15560)

## Background

Gemini 3 rejects replayed assistant `functionCall` parts without
`thoughtSignature` with HTTP 400. This happens when app/client code
persists or rebuilds messages and drops `providerOptions`.

## Summary

- For Gemini 3 only (`/^gemini-3[.-]/`), inject Google's documented
`skip_thought_signature_validator` sentinel when a replayed tool call
has no signature.
- Warn once per request with affected tool names.
- Adapted to `main` V4 Google provider files and existing `google` /
`googleVertex` / `vertex` namespace lookup.

## Manual Verification

Live replay against `google('gemini-3-flash-preview')`:
missing-signature replay changed from HTTP 400 to HTTP 200 with warning.

## Checklist

- [ ] All commits are signed (PRs with unsigned commits cannot be
merged)
- [x] Tests have been added / updated (for bug fixes / features)
- [ ] Documentation has been added / updated (for bug fixes / features)
- [x] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] I have reviewed this pull request (self-review)

## Related Issues

Closes #15550.
Refs #10344, #11413, #14196, #15548, #13060.

Co-authored-by: Cursor <cursoragent@cursor.com>
gr2m added a commit that referenced this pull request May 27, 2026
… tool-call replays (#15553)

## Background

Gemini 3 rejects replayed assistant `functionCall` parts without
`thoughtSignature` with HTTP 400. This happens when app/client code
persists or rebuilds messages and drops `providerOptions`.

## Summary

- For Gemini 3 only (`/^gemini-3[.-]/`), inject Google's documented
`skip_thought_signature_validator` sentinel when a replayed tool call
has no signature.
- Warn once per request with affected tool names.
- Centralize signature namespace lookup and honor `google`,
`googleVertex`, and `vertex` fallbacks.

## Manual Verification

Live replay against `google('gemini-3-flash-preview')`:
missing-signature replay changed from HTTP 400 to HTTP 200 with warning.

## Checklist

- [x] Tests have been added / updated (for bug fixes / features)
- [ ] Documentation has been added / updated (for bug fixes / features)
- [x] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] I have reviewed this pull request (self-review)

## Related Issues

Closes #15550.
Refs #10344, #11413, #14196, #15548, #13060.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant