Gemini thought summaries should not be sent back to API if there are thought signatures

[lwnz42]

Question

With gemini models, thought summaries are developer / user facing summaries of model thought process. They are to be used for debugging / display purposes for end users or developers.

As per google gemini documentation.

Thought summaries are synthesized versions of the model's raw thoughts and offer insights into the model's internal reasoning process.

They serve different purpose than raw reasoning text from models like GPT, for example (which OpenAI recommends to pass back to model AFAIK).

But Gemini models have thought signatures for that, which Pydantic AI is handling already - those are needed for preserving the reasoning context and for tool calling.

I understand saving them to message history (so developers can still capture those and show them if they're running user-facing chat-like apps), but why is pydantic AI not skipping the summaries when mapping the past context into parts for google models?

These can be quite verbose and pollute the context window unnecessarily.

I understand that one of main pydantic AI features is being able to use any model with native API and have unified interface, so I'm not sure if that's an oversight (because some other models recommend adding thoughts back), or a feature for multi-provider compatibility (like if the history is being passed from other model, which generates raw thoughts as text).

Additional Context

No response

[DouweM]

why is pydantic AI not skipping the summaries when mapping the past context into parts for google models?

@lwnz42 Do I understand correctly that you're suggesting skipping setting the text field here, and only sending the thought signature?

pydantic-ai/pydantic_ai_slim/pydantic_ai/models/google.py

Lines 998 to 1005 in 8015eb7

	elif isinstance(item, ThinkingPart):
	if item.provider_name == provider_name and item.signature:
	# The thought signature is to be included on the _next_ part, not the thinking part itself
	thinking_part_signature = item.signature
	if item.content:
	part['text'] = item.content
	part['thought'] = True

These can be quite verbose and pollute the context window unnecessarily.

When we send the thought summaries back to the Gemini API as we received them, along with the thought signature, I assume that they are not inserted into the context window, but rather the original thoughts (derived from the signature). So it increases the API request size, but I don't think it'd affect the context window whether we sent the summary text or not. (If we can verify that Google behaves the same with or without the summary text, I agree we can drop it to save on API request size)

I understand that one of main pydantic AI features is being able to use any model with native API and have unified interface, so I'm not sure if that's an oversight (because some other models recommend adding thoughts back), or a feature for multi-provider compatibility (like if the history is being passed from other model, which generates raw thoughts as text).

The reason the summaries are stored on the message history is primarily so it can be shown to users, as you identified, but indeed we also send them when history from one model is shared with another, assuming it'll perform (i.e. "pick up where the other model left off") better when it has more context, including things that made it into the thinking summary but not the output text. (If that's undesirable, you could filter them out using https://ai.pydantic.dev/message-history/#processing-message-history)

[lwnz42]

Do I understand correctly that you're suggesting skipping setting the text field here, and only sending the thought signature?

Yes, but I realize now this could be a "gemini only" solution, if you want thinking traces to be present when switching models within same message history in at least some form (because signatures from one model will probably not work as valid reasoning chain for another model).

But I'll elaborate just to make sure I explained what I meant.

When we send the thought summaries back to the Gemini API as we received them, along with the thought signature, I assume that they are not inserted into the context window, but rather the original thoughts (derived from the signature).

pydantic-ai/pydantic_ai_slim/pydantic_ai/models/google.py

Lines 1065 to 1073 in 0b279de

	for part in parts:
	provider_details: dict[str, Any] | None = None
	if part.thought_signature:
	# Per https://ai.google.dev/gemini-api/docs/function-calling?example=meeting#thought-signatures:
	# - Always send the thought_signature back to the model inside its original Part.
	# - Don't merge a Part containing a signature with one that does not. This breaks the positional context of the thought.
	# - Don't combine two Parts that both contain signatures, as the signature strings cannot be merged.
	thought_signature = base64.b64encode(part.thought_signature).decode('utf-8')
	provider_details = {'thought_signature': thought_signature}

The thought signatures are saved on the original part here, so any part that has thought_signature field gets its' signature added to the provider_details field.

Then here:

pydantic-ai/pydantic_ai_slim/pydantic_ai/models/google.py

Lines 1081 to 1088 in 0b279de

	elif part.text is not None:
	# Google sometimes sends empty text parts, we don't want to add them to the response
	if len(part.text) == 0 and not provider_details:
	continue
	if part.thought:
	item = ThinkingPart(content=part.text)
	else:
	item = TextPart(content=part.text)

You create a thinking part based on "part": "text" and "thought": True flag.

These ThinkingParts will never have thought signature in Gemini models.

Since thought signatures are added on last part in the response. "part": "text", "thought": True parts will never be last part of the response, because response is always either ending in text part that is not thought, or function call, or inline data (I didn't test multimodal though, only text responses).

So right now you send both, but only need signature passed back.

About API behaviour:

I assume that they are not inserted into the context window, but rather the original thoughts (derived from the signature)

At least on genai SDK level nothing special is happening with summaries, they are treated as part and passed back to API, not filtered

lines 3000-3001 in models.py of python genai SDK.

  if getv(from_object, ['thought']) is not None:
      setv(to_object, ['thought'], getv(from_object, ['thought']))

I can try to search the REST API reference to see if there's more information, but in their notebook examples they don't send the summaries back to model in the request body ( https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/thinking/intro_thought_signatures_rest.ipynb).

In step 1 response shows both a function call part AND a thought summary part, but when sending back in step 2, only the function call with signature is included - the thought text is dropped.

[DouweM]

These ThinkingParts will never have thought signature in Gemini models.

@lwnz42 Good point, we changed that when Gemini 3 Pro came out: we used to always store the thought signatures on the preceding ThinkingPart, but now we store them on the final part that they came on, so I was commenting based on an outdated memory :) My point was more that we send everything back verbatim, without framework-side filtering of summaries.

At least on genai SDK level nothing special is happening with summaries, they are treated as part and passed back to API, not filtered

Right, I was expecting it to be the API itself that filters them out instead of passing the summaries into the LLM.

As far as I'm aware, the expectation with all the LLM APIs is that the exact response parts that are received in one step can be sent back as request parts in the next step, which is what we're doing here. But if their API accepts it (and we can verify that it doesn't affect performance), I'm OK with not sending back the summaries, although I think it'd only affect the API payload size, not the actual LLM context window.

in their notebook examples they don't send the summaries back to model in the request body ( GoogleCloudPlatform/generative-ai@main/gemini/thinking/intro_thought_signatures_rest.ipynb).

I suggest checking what the Google ADK does. I'd imagine it sends the response parts back verbatim in the request, but if not, we should copy what they filter.

[lwnz42]

@DouweM

tldr: I had AI parse the ADK code, it does skip the summaries.

1. Parts with thought=True are filtered out when building content to send to the API with generate content API.

In flows/llm_flows/contents.py:222-232:

def _is_part_invisible(p: types.Part) -> bool:
  """Returns whether a part is invisible for LLM context."""
  return getattr(p, 'thought', False) or not (
      p.text
      or p.inline_data
      or p.file_data
      or p.function_call
      or p.function_response
      or p.executable_code
      or p.code_execution_result
  )

2. When converting to Interactions API format - only signature is sent

In models/interactions_utils.py:149-157:

elif part.thought:
    # part.thought is a boolean indicating this is a thought part
    # ThoughtContentParam expects 'signature' (base64 encoded bytes)
    result: dict[str, Any] = {'type': 'thought'}
    if part.thought_signature is not None:
      result['signature'] = base64.b64encode(part.thought_signature).decode('utf-8')
    return result

No text summary - only {'type': 'thought', 'signature': ...}

3. When receiving thought outputs from API - they're skipped

In models/interactions_utils.py:361-365:

elif output_type == 'thought':
    # ThoughtContent has a 'signature' attribute, not 'thought'
    # These are internal model reasoning and typically not exposed as Parts
    # Skip thought outputs for now
    return None

4. Thoughts excluded from multi-agent context explicitly

In flows/llm_flows/contents.py:521-527:

for part in event.content.parts:
    if part.thought:
      # Exclude thoughts from the context.
      continue

---

I also checked your assumption that API filters them, I figured if it does - it won't count towards input tokens on next request when sending past context back, right?

So. test 1: request with thought parts (thought=True + text) and test 2: request without thought parts.

Test	Input Tokens
WITH thought parts	178
WITHOUT thought parts	17
Difference	161 extra tokens

So Gemini API doesn't filter them out, since they're included and billed as input tokens.

---

The question about cross-model context is still open - use cases like:

agent1 (google model) > handoff to agent2 (openAI model) with message history from agent1.

Would lose any reasoning traces, even in the form of a summary, if you ignore thinking summaries, since I don't think signatures work the same across models (or do they? I don't actually know).

and vice versa:

if we have ThinkingPart ignored on model level, you won't be able to get the raw reasoning text from models that do generate it when google model is constructing the contents.

But more and more models today hide the raw thoughts behind signatures so... Not sure about this, you probably have more insight on the topic, I've been mostly using gemini models only for my projects in the past.

[DouweM]

@lwnz42 Great findings, thank you! I'll double check with our contacts at Google, but I think the right move will be to not send ThinkingPart.content to Google that was generated by Google (by storing/checking ThinkingPart.provider_name), as we know they're unnecessary (and actively harmful to the context window) when we already send signatures.

So. test 1: request with thought parts (thought=True + text) and test 2: request without thought parts.

Can you share the code you used to test that please? I just want to verify for myself 😄

[lwnz42]

@DouweM

No worries, happy it turned out something useful.

Here's the code

thought_summary_token_bug.py

[DouweM]

@lwnz42 Note that my assumption was that thought summaries get ignored by the API if the original response with thought_signatures is sent back verbatim, which is not the case in your example because:

1. gemini-2.5-flash-lite doesn't generate thought signatures
2. we're not sending back the full response.candidates[0].content but rather only text parts with/without thought=True.

So I wouldn't be surprised if my assumption is still wrong (and our contacts at Google should be able to verify that), but that script doesn't prove it 😄