Support reasoning options

[simonw]

https://openrouter.ai/docs/use-cases/reasoning-tokens#controlling-reasoning-tokens

You can control reasoning tokens in your requests using the reasoning parameter:

{
  "model": "your-model",
  "messages": [],
  "reasoning": {
    // One of the following (not both):
    "effort": "high", // Can be "high", "medium", or "low" (OpenAI-style)
    "max_tokens": 2000, // Specific token limit (Anthropic-style)
    // Optional: Default is false. All models support this.
    "exclude": false, // Set to true to exclude reasoning tokens from response
    // Or enable reasoning with the default parameters:
    "enabled": true // Default: inferred from `effort` or `max_tokens`
  }
}

[simonw]

I'm going to add these as options called reasoning_effort, reasoning_max_tokens, reasoning_exclude and reasoning_enabled.

[simonw]

Tried it like this:

llm -m openrouter/openai/gpt-5 'say hello'
llm -m openrouter/openai/gpt-5 'say hello' -o reasoning_effort low
llm -m openrouter/openai/gpt-5 'say hello' -o reasoning_effort medium
llm -m openrouter/openai/gpt-5 'say hello' -o reasoning_effort high

Then llm logs -n 4 --json to see this:

[
  {
    "id": "01k5mfm5pk1gn18v5tcsf09zdd",
    "model": "openrouter/openai/gpt-5",
    "prompt": "say hello",
    "system": null,
    "prompt_json": {
      "messages": [
        {
          "role": "user",
          "content": "say hello"
        }
      ]
    },
    "options_json": {},
    "response": "Hello! How can I help you today?",
    "response_json": {
      "content": {
        "$": "r:01k5mfm5pk1gn18v5tcsf09zdd"
      },
      "role": "assistant",
      "finish_reason": "stop",
      "usage": {
        "completion_tokens": 79,
        "prompt_tokens": 8,
        "total_tokens": 87,
        "completion_tokens_details": {
          "reasoning_tokens": 64,
          "image_tokens": 0
        },
        "prompt_tokens_details": {
          "audio_tokens": 0,
          "cached_tokens": 0
        },
        "cost": 0.0004,
        "is_byok": false,
        "cost_details": {
          "upstream_inference_prompt_cost": 5e-06,
          "upstream_inference_completions_cost": 0.000395
        }
      },
      "id": "gen-1758402840-SX37MEvUpwdoNKzE81ED",
      "object": "chat.completion.chunk",
      "model": "openai/gpt-5",
      "created": 1758402840
    },
    "conversation_id": "01k5mfm5phg03n8y1qzkxe3a1e",
    "duration_ms": 4400,
    "datetime_utc": "2025-09-20T21:13:59.969225+00:00",
    "input_tokens": 8,
    "output_tokens": 79,
    "token_details": "{\"completion_tokens_details\": {\"reasoning_tokens\": 64}, \"cost\": 0.0004, \"cost_details\": {\"upstream_inference_prompt_cost\": 5e-06, \"upstream_inference_completions_cost\": 0.000395}}",
    "conversation_name": "say hello",
    "conversation_model": "openrouter/openai/gpt-5",
    "schema_json": null,
    "prompt_fragments": [],
    "system_fragments": [],
    "attachments": []
  },
  {
    "id": "01k5mfmef45cqchfmbzgbjk4em",
    "model": "openrouter/openai/gpt-5",
    "prompt": "say hello",
    "system": null,
    "prompt_json": {
      "messages": [
        {
          "role": "user",
          "content": "say hello"
        }
      ]
    },
    "options_json": {
      "reasoning_effort": "low"
    },
    "response": "Hello! How can I help you today?",
    "response_json": {
      "content": {
        "$": "r:01k5mfmef45cqchfmbzgbjk4em"
      },
      "role": "assistant",
      "finish_reason": "stop",
      "usage": {
        "completion_tokens": 79,
        "prompt_tokens": 8,
        "total_tokens": 87,
        "completion_tokens_details": {
          "reasoning_tokens": 64,
          "image_tokens": 0
        },
        "prompt_tokens_details": {
          "audio_tokens": 0,
          "cached_tokens": 0
        },
        "cost": 0.0004,
        "is_byok": false,
        "cost_details": {
          "upstream_inference_prompt_cost": 5e-06,
          "upstream_inference_completions_cost": 0.000395
        }
      },
      "id": "gen-1758402848-7hsyEUCQV3p7KgaAYuL9",
      "object": "chat.completion.chunk",
      "model": "openai/gpt-5",
      "created": 1758402848
    },
    "conversation_id": "01k5mfmef1jjv2fmfgehgns2pr",
    "duration_ms": 5296,
    "datetime_utc": "2025-09-20T21:14:08.048657+00:00",
    "input_tokens": 8,
    "output_tokens": 79,
    "token_details": "{\"completion_tokens_details\": {\"reasoning_tokens\": 64}, \"cost\": 0.0004, \"cost_details\": {\"upstream_inference_prompt_cost\": 5e-06, \"upstream_inference_completions_cost\": 0.000395}}",
    "conversation_name": "say hello",
    "conversation_model": "openrouter/openai/gpt-5",
    "schema_json": null,
    "prompt_fragments": [],
    "system_fragments": [],
    "attachments": []
  },
  {
    "id": "01k5mfmmnjergv50xqeeg8hedg",
    "model": "openrouter/openai/gpt-5",
    "prompt": "say hello",
    "system": null,
    "prompt_json": {
      "messages": [
        {
          "role": "user",
          "content": "say hello"
        }
      ]
    },
    "options_json": {
      "reasoning_effort": "medium"
    },
    "response": "Hello!",
    "response_json": {
      "content": {
        "$": "r:01k5mfmmnjergv50xqeeg8hedg"
      },
      "role": "assistant",
      "finish_reason": "stop",
      "usage": {
        "completion_tokens": 8,
        "prompt_tokens": 8,
        "total_tokens": 16,
        "completion_tokens_details": {
          "reasoning_tokens": 0,
          "image_tokens": 0
        },
        "prompt_tokens_details": {
          "audio_tokens": 0,
          "cached_tokens": 0
        },
        "cost": 4.5e-05,
        "is_byok": false,
        "cost_details": {
          "upstream_inference_prompt_cost": 5e-06,
          "upstream_inference_completions_cost": 4e-05
        }
      },
      "id": "gen-1758402856-CeDR3nXAC01JrWo9ZsWe",
      "object": "chat.completion.chunk",
      "model": "openai/gpt-5",
      "created": 1758402856
    },
    "conversation_id": "01k5mfmmnhxhytjhf2egvd1rad",
    "duration_ms": 3262,
    "datetime_utc": "2025-09-20T21:14:16.434798+00:00",
    "input_tokens": 8,
    "output_tokens": 8,
    "token_details": "{\"cost\": 4.5e-05, \"cost_details\": {\"upstream_inference_prompt_cost\": 5e-06, \"upstream_inference_completions_cost\": 4e-05}}",
    "conversation_name": "say hello",
    "conversation_model": "openrouter/openai/gpt-5",
    "schema_json": null,
    "prompt_fragments": [],
    "system_fragments": [],
    "attachments": []
  },
  {
    "id": "01k5mfmv0v7c6n6k6yj2nssbz0",
    "model": "openrouter/openai/gpt-5",
    "prompt": "say hello",
    "system": null,
    "prompt_json": {
      "messages": [
        {
          "role": "user",
          "content": "say hello"
        }
      ]
    },
    "options_json": {
      "reasoning_effort": "high"
    },
    "response": "Hello!",
    "response_json": {
      "content": {
        "$": "r:01k5mfmv0v7c6n6k6yj2nssbz0"
      },
      "role": "assistant",
      "finish_reason": "stop",
      "usage": {
        "completion_tokens": 72,
        "prompt_tokens": 8,
        "total_tokens": 80,
        "completion_tokens_details": {
          "reasoning_tokens": 64,
          "image_tokens": 0
        },
        "prompt_tokens_details": {
          "audio_tokens": 0,
          "cached_tokens": 0
        },
        "cost": 0.000365,
        "is_byok": false,
        "cost_details": {
          "upstream_inference_prompt_cost": 5e-06,
          "upstream_inference_completions_cost": 0.00036
        }
      },
      "id": "gen-1758402863-OliHs4P7kAMHc04kcs1n",
      "object": "chat.completion.chunk",
      "model": "openai/gpt-5",
      "created": 1758402863
    },
    "conversation_id": "01k5mfmv0s5fjpdmc0a4ts518a",
    "duration_ms": 2874,
    "datetime_utc": "2025-09-20T21:14:23.327100+00:00",
    "input_tokens": 8,
    "output_tokens": 72,
    "token_details": "{\"completion_tokens_details\": {\"reasoning_tokens\": 64}, \"cost\": 0.000365, \"cost_details\": {\"upstream_inference_prompt_cost\": 5e-06, \"upstream_inference_completions_cost\": 0.00036}}",
    "conversation_name": "say hello",
    "conversation_model": "openrouter/openai/gpt-5",
    "schema_json": null,
    "prompt_fragments": [],
    "system_fragments": [],
    "attachments": []
  }
]

Weird that three of them used 64 reasoning tokens and one used 0 reasoning tokens. I'm not sure this is working correctly.

[simonw]

Trying something a bit more interesting:

llm -m openrouter/openai/gpt-5 'convince me that dogs are real'
llm -m openrouter/openai/gpt-5 'convince me that dogs are real' -o reasoning_effort low
llm -m openrouter/openai/gpt-5 'convince me that dogs are real' -o reasoning_effort medium
llm -m openrouter/openai/gpt-5 'convince me that dogs are real' -o reasoning_effort high

Then I counted reasoning tokens:

llm logs -n 4 --json | jq '[.[].response_json.usage.completion_tokens_details]'

Output:

[
  {
    "reasoning_tokens": 1152,
    "image_tokens": 0
  },
  {
    "reasoning_tokens": 512,
    "image_tokens": 0
  },
  {
    "reasoning_tokens": 1024,
    "image_tokens": 0
  },
  {
    "reasoning_tokens": 3328,
    "image_tokens": 0
  }
]

So it looked like that worked. Full output here: https://gist.github.com/simonw/1135fb64c1c66ca3dd649e0f7d2e791d#file-dogs-md

[simonw]

Apparently GPT-5 on OpenRouter does not allow reasoning_enabled: false:

llm -m openrouter/openai/gpt-5 'convince me that dogs are real' -o reasoning_enabled false
Error: Error code: 400 - {'error': {'message': 'Reasoning is mandatory for this endpoint and cannot be disabled.', 'code': 400, 'metadata': {'provider_name': None}}}

[simonw]

Pushed my code so far, most relevant piece is this:

llm-openrouter/llm_openrouter.py

Lines 49 to 64 in 8ffae50

	reasoning_effort: Optional[ReasoningEffortEnum] = Field(
	description='One of "high", "medium", or "low" to control reasoning effort',
	default=None,
	)
	reasoning_max_tokens: Optional[int] = Field(
	description="Specific token limit to control reasoning effort",
	default=None,
	)
	reasoning_exclude: Optional[bool] = Field(
	description="Set to true to exclude reasoning tokens from response",
	default=None,
	)
	reasoning_enabled: Optional[bool] = Field(
	description="Set to true to enable reasoning with default parameters",
	default=None,
	)

reasoning_exclude doesn't seem to do anything useful yet, because I don't think reasoning tokens are displayed at all right now.

[simonw]

I ran llm install -U openai and everything broke:

NameError: Fields must not use names with leading underscores; e.g., use 'pydantic_extra__' instead of '__pydantic_extra__'.

I think because it upgraded Pydantic.

[simonw]

LLM_RAISE_ERRORS=1 python -i -m llm -m openrouter/openai/gpt-5 'convince me that dogs are real' -o reasoning_effort high

NameError: Fields must not use names with leading underscores; e.g., use 'pydantic_extra__' instead of '__pydantic_extra__'.
>>> import pdb; pdb.pm()
> /Users/simon/.local/share/virtualenvs/llm-openrouter-ZELPBBnK/lib/python3.12/site-packages/pydantic/_internal/_model_construction.py(381)inspect_namespace()
-> raise NameError(
(Pdb) list
376  	                )
377  	            private_attributes[var_name] = value
378  	            del namespace[var_name]
379  	        elif isinstance(value, FieldInfo) and not is_valid_field_name(var_name):
380  	            suggested_name = var_name.lstrip('_') or 'my_field'  # don't suggest '' for all-underscore name
381  ->	            raise NameError(
382  	                f'Fields must not use names with leading underscores;'
383  	                f' e.g., use {suggested_name!r} instead of {var_name!r}.'
384  	            )
385  	
386  	        elif var_name.startswith('__'):
(Pdb) var_name
'__pydantic_extra__'

[simonw]

It's this issue:

• NameError: Incompatibilities with older Pydantic v2 versions causing NameError when importing from openai openai/openai-python#2644

I ran this to fix it, but this means I probably need to think about this dependency elsewhere in LLM:

pip install pydantic==2.11.9

[simonw]

I ran this:

LLM_OPENAI_SHOW_RESPONSES=1 \
  llm -m openrouter/openai/gpt-5 'describe dogs in a surprising way' -o reasoning_effort high

Which showed me why the reasoning text is not displayed:

data: {"id":"gen-1758404056-c5lcrC98mCQEmoX68Gkj","provider":"OpenAI","model":"openai/gpt-5","object":"chat.completion.chunk","created":1758404056,"choices":[{"index":0,"delta":{"role":"assistant","content":"","reasoning":"ell","reasoning_details":[{"type":"reasoning.summary","summary":"ell","format":"openai-responses-v1","index":0}]},"finish_reason":null,"native_finish_reason":null,"logprobs":null}]}


data: {"id":"gen-1758404056-c5lcrC98mCQEmoX68Gkj","provider":"OpenAI","model":"openai/gpt-5","object":"chat.completion.chunk","created":1758404056,"choices":[{"index":0,"delta":{"role":"assistant","content":"","reasoning":" archae","reasoning_details":[{"type":"reasoning.summary","summary":" archae","format":"openai-responses-v1","index":0}]},"finish_reason":null,"native_finish_reason":null,"logprobs":null}]}

Key bit:

"choices":[{"index":0,"delta":{"role":"assistant","content":"","reasoning":" archae","reasoning_details":[{"type":"reasoning.summary","summary":" archae","format":"openai-responses-v1","index":0}]},"finish_reason":null,"native_finish_reason":null,"logprobs":null}]}

Note "content":"","reasoning":"archae" - so the content key is empty and there's a reasoning key to get output from instead.

[simonw]

For the moment I'm going to drop reasoning_exclude entirely, I can add that again when I solve this in LLM core:

• Annotations abstraction for responses that are not just a stream of plain text llm#716