Thinking models are unreliable when `max_output_tokens` set due to them ignoring the thinking budget

[ascillitoe]

Three different issues make the thinking models (2.5 flash and pro) unreliable:

1. The thinking_budget appears to be ignored. We get non-zero (and large) thoughts_token_count values regardless of the thinking_budget config. This appears to be a known issue?
2. A MAX_TOKENS finish reason appears to be given if thoughts_token_count + output_token_count > max_output_tokens. This is problematic since we don't appear to have proper control over thoughts_token_count right now.
3. If MAX_TOKENS finish reason is triggered, the response text is empty, making debugging of the above very difficult (we can't get a sense of the size of the output text).

We only started experiencing consistent MAX_TOKENS responses sometime last week, so it seems something might have changed recently? Either the default thinking budget, or the way max_output_tokens is applied?

The only solution that currently works for us is to set max_output_tokens=None, but then the thinking tokens often increase to ~6k even for very simple tasks.

Edit: we have only seen this occur when using structured output. I'm can't confirm if it occurs for unstructured output.

Environment details

• Programming language: Python
• OS: Ubuntu 22.04
• Language runtime version: 3.10
• Package version: 1.14.0

Example

The docs suggest setting thinking_budget=0 will turn thinking off, however with the following request:

class RelevantDataFields(BaseModel):
    fields: List[Literal[tuple(field_names)]]

model_config = GenerateContentConfig(
    system_instruction=self.system_instruction,
    thinking_config=ThinkingConfig(thinking_budget=0),
    max_output_tokens=2000,
    temperature=0.,
    response_schema=RelevantDataFields.model_json_schema(),
    response_mime_type='application/json'
)
pred = await aio.models.generate_content(
    model='gemini-2.5-pro-preview-03-25',
    contents=prompt,
    config=model_config
)

the response is:

GenerateContentResponse(candidates=[Candidate(content=Content(parts=[Part(video_metadata=None,
thought=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None,
function_response=None, inline_data=None, text='')], role='model'), citation_metadata=None,
finish_message=None, token_count=None, finish_reason=<FinishReason.MAX_TOKENS: 'MAX_TOKENS'>,
avg_logprobs=None, grounding_metadata=None, index=0, logprobs_result=None, safety_ratings=None)],
create_time=None, response_id=None, model_version='models/gemini-2.5-pro-preview-05-06',
prompt_feedback=None,
usage_metadata=GenerateContentResponseUsageMetadata(cache_tokens_details=None,
cached_content_token_count=None, candidates_token_count=None, candidates_tokens_details=None,
prompt_token_count=1674, prompt_tokens_details=[ModalityTokenCount(modality=<MediaModality.TEXT: 'TEXT'>, token_count=1674)], thoughts_token_count=2000, tool_use_prompt_token_count=None, tool_use_prompt_tokens_details=None, total_token_count=3674, traffic_type=None), automatic_function_calling_history=[], parsed=None)

The thoughts_token_count=2000 despite setting thinking_budget=0, and the model version is also wrong (05-06 instead of 03-25).

Ideal solution

1. thinking_budget should constrain thoughts_token_count
2. MAX_TOKENS finish reason should only be returned if the number of output tokens is greater than max_output_tokens, rather than including the thinking budget (assuming this is the case?).
3. The computed output so far should be returned if MAX_TOKENS is reached.

[ascillitoe]

Update: I've just noticed when I get FinishReason.MaxTokens, thoughts_token_count = max_output_tokens, so the thought tokens definitely appear to be linked to the output token limit now? For example below is a response with max_output_tokens=5000 and thinking_budget=0:

GenerateContentResponse(candidates=[Candidate(content=Content(parts=[Part(video_metadata=None, thought=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=None, inline_data=None, text='')], role='model'), citation_metadata=None, finish_message=None, token_count=None, finish_reason=<FinishReason.MAX_TOKENS: 'MAX_TOKENS'>, avg_logprobs=None, grounding_metadata=None, index=0, logprobs_result=None, safety_ratings=None)], create_time=None, response_id=None, model_version='models/gemini-2.5-flash-preview-04-17', prompt_feedback=None, usage_metadata=GenerateContentResponseUsageMetadata(cache_tokens_details=None, cached_content_token_count=None, candidates_token_count=None, candidates_tokens_details=None, prompt_token_count=4725, prompt_tokens_details=[ModalityTokenCount(modality=<MediaModality.TEXT: 'TEXT'>, token_count=4725)], thoughts_token_count=5000, tool_use_prompt_token_count=None, tool_use_prompt_tokens_details=None, total_token_count=9725, traffic_type=None), automatic_function_calling_history=[], parsed=None)

This is for a task that, before last week, ran very reliably with max_output_tokens=500 on gemini-2.5-pro-preview-03-25, so something has changed recently.

[sararob]

Thank you for this feedback, I've passed it along to the API team and will let you know when we have an update.

[ascillitoe]

Thanks @sararob. Just as an FYI, I missed out a small but potentially important bit of information earlier. We've only seen this issue occur when using structured output. I've updated my initial post to mention this.

[MiguelLopezVirues]

Hi, I have the same problem here, using the response_mime_type = "text/plain". I hit the MAX.TOKEN limit due to the thoughts_token_count.

Below the response.model_dump() (Note that here I do not limit the output tokens, so the limit is effectively 65536):

{'candidates': [{'content': {'parts': [{'video_metadata': None,
      'thought': None,
      'code_execution_result': None,
      'executable_code': None,
      'file_data': None,
      'function_call': None,
      'function_response': None,
      'inline_data': None,
      'text': ''}],
    'role': 'model'},
   'citation_metadata': None,
   'finish_message': None,
   'token_count': None,
   'finish_reason': <FinishReason.MAX_TOKENS: 'MAX_TOKENS'>,
   'avg_logprobs': None,
   'grounding_metadata': None,
   'index': 0,
   'logprobs_result': None,
   'safety_ratings': None}],
 'create_time': None,
 'response_id': None,
 'model_version': 'models/gemini-2.5-flash-preview-04-17',
 'prompt_feedback': None,
 'usage_metadata': {'cache_tokens_details': None,
  'cached_content_token_count': None,
  'candidates_token_count': None,
  'candidates_tokens_details': None,
  'prompt_token_count': 42457,
  'prompt_tokens_details': [{'modality': <MediaModality.TEXT: 'TEXT'>,
    'token_count': 661},
   {'modality': <MediaModality.DOCUMENT: 'DOCUMENT'>, 'token_count': 41796}],
  'thoughts_token_count': 65536,
  'tool_use_prompt_token_count': None,
  'tool_use_prompt_tokens_details': None,
  'total_token_count': 107993,
  'traffic_type': None},
 'automatic_function_calling_history': [],
 'parsed': None}

[aeneaswiener]

We noticed this too. thinking_budget appears to be respected in AI Sduio, but not to the same way on the SDK here

To reproduce, run prompt

give me a hard math problem and solve it

with thinking_budget 0, 50, 1000, 2000

It is perfectly respected in AI Studio for all these values

Gen AI SDK only respects 0, 1000, 2000, but when setting to 50, the actual thinking tokens used are closer to 1000 (for the same prompt that works fine in AI Sdudio

[aeneaswiener]

Update, i have realised this issue is due to the authentication used. Token budget is respected when using API key auth, and is not respected when using vertex authentication with service default creds.

To reproduce

from google import genai
from google.genai import types as genai_types
import os

# thinking_budget=50 is respected with this client, thoughts_token_count is 21
client = genai.Client(api_key="TBC")

# thinking_budget=50 is NOT respected with this client, thoughts_token_count is 700
client = genai.Client(
    vertexai=True, project='TBC', location='us-central1'
)

response = await client.aio.models.generate_content(
    model="gemini-2.5-flash-preview-04-17",
    contents="Explain how AI works in a few words?",
    config=genai_types.GenerateContentConfig(
        thinking_config=genai_types.ThinkingConfig(
            thinking_budget=50,
        )
    )
)
print(response.text)
print(response.usage_metadata.thoughts_token_count)

[ascillitoe]

That is strange, we use api_key and consistently don't get the thinking_budget respected with gemini-2.5-pro-preview. The problem is bad enough for us that we've had to migrate to other providers for now.

[ascillitoe]

Thank you for this feedback, I've passed it along to the API team and will let you know when we have an update.

Hi @sararob , has the API team made any progress on this?

[pamorgan]

API team addressed some of the raised concerns with include_thoughts and thinking_budget=0.
Please refer to documentation in https://ai.google.dev/gemini-api/docs/thinking and example in https://github.com/google-gemini/cookbook/blob/main/quickstarts/Get_started_thinking.ipynb

Please let us know if the issue persisted.

[ascillitoe]

Hi @pamorgan we just tried again, we still have the same problem as before with structured generation and gemini-2.5-pro-preview-05-06. We set thinking_budget=0, but we always get an empty response because we run out of output tokens (since thoughts_token_count = max_output_tokens).

[MiguelLopezVirues]

In addition to the whole concern of gemini-2.5-flash not respecting thinking budgets, there's the fact that thoughts-token-counts are not included within the outputs field of the response object. If that is the case, even with issues respecting thought-budget we shouldn't be bumping into MAX.TOKENS problem. I believe to have seen documentation that supports my statement.

Additionally, thought-budget allows 24k tokens max, so the responses shouldn't in any case surpass that up to the 65k output limit. That must mean that even fixed internal limits are not respected :/

Anyway, thank you very much for the support @pamorgan @sararob :)