Skip to content

fix(models): send AnyLLM Responses reasoning as a mapping - #4138

Merged
seratch merged 1 commit into
openai:mainfrom
hsusul:fix/any-llm-responses-reasoning-payload
Aug 3, 2026
Merged

fix(models): send AnyLLM Responses reasoning as a mapping#4138
seratch merged 1 commit into
openai:mainfrom
hsusul:fix/any-llm-responses-reasoning-payload

Conversation

@hsusul

@hsusul hsusul commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

AnyLLMModel on the Responses path sends ModelSettings.reasoning as a list of [key, value] pairs instead of a mapping, so every AnyLLM Responses request that configures reasoning fails with a pydantic ValidationError before it reaches the provider.

_fetch_responses_response builds the payload with:

"reasoning": _to_dump_compatible(model_settings.reasoning) if model_settings.reasoning is not None else None,

_to_dump_compatible exists to materialize lazy iterables (#1683, #3700). It has no BaseModel branch, and a pydantic model is collections.abc.Iterable — it iterates as key/value pairs — so it falls through to the generic iterable branch:

>>> from openai.types.shared import Reasoning
>>> from agents.util._json import _to_dump_compatible
>>> _to_dump_compatible(Reasoning(effort="low", summary="concise"))
[['context', None], ['effort', 'low'], ['generate_summary', None], ['mode', None], ['summary', 'concise']]

any-llm types ResponsesParams.reasoning as dict[str, Any] | None, and _call_any_llm_responses constructs that model, so the request dies at validation:

pydantic_core._pydantic_core.ValidationError: 1 validation error for ResponsesParams
reasoning
  Input should be a valid dictionary [type=dict_type, input_value=[['context', None], ['eff... ['summary', 'concise']], input_type=list]

This is not a rare path. _build_responses_transport_kwargs always sets a User-Agent header, so transport_kwargs is never empty and _call_any_llm_responses always takes the ResponsesParams branch. Reasoning is therefore unusable on the AnyLLM Responses path for both get_response and stream_response, which share this payload builder. Present in v0.19.2 and unchanged since the adapter landed in #2706.

Minimal reproduction

No API key or network call — the failure happens while building the request. Requires the optional any-llm-sdk extra.

import asyncio
from openai.types.shared import Reasoning
from agents import ModelSettings, ModelTracing
from agents.extensions.models import any_llm_model

class FakeProvider:
    SUPPORTS_RESPONSES = True
    async def _aresponses(self, params, **kwargs): return None

any_llm_model.AnyLLM = type("F", (), {"create": staticmethod(lambda *a, **k: FakeProvider())})
model = any_llm_model.AnyLLMModel(model="openai/gpt-5.4-mini")

asyncio.run(model.get_response(
    system_instructions=None, input="hi",
    model_settings=ModelSettings(reasoning=Reasoning(effort="low", summary="concise")),
    tools=[], output_schema=None, handoffs=[], tracing=ModelTracing.DISABLED,
    previous_response_id=None, conversation_id=None, prompt=None,
))
# ValidationError: reasoning -> Input should be a valid dictionary

Current behavior: ValidationError; the request never reaches the provider.
Corrected behavior: params.reasoning == {"effort": "low", "summary": "concise"} and the request is sent.

Implementation

One call site, one line: dump the model instead of routing it through the iterable materializer.

"reasoning": model_settings.reasoning.model_dump(mode="json", exclude_none=True)

ModelSettings.reasoning is typed and validated as Reasoning | None (a dict passed by a user is coerced to Reasoning at construction), so model_dump is the source-of-truth conversion. mode="json" matches the existing ItemHelpers.copy_tool_call_caller convention, and exclude_none=True matches this adapter's own _sanitize_any_llm_responses_value, which strips None from every payload it hands to any-llm.

Why this is minimal: the demonstrated defect is one call site misusing a lazy-iterable materializer for a pydantic model. Broadening _to_dump_compatible to special-case BaseModel would change tool, message, and replay serialization across the OpenAI Responses, Chat Completions, LiteLLM, run-state, and rollout paths with no demonstrated failure in any of them, so that is an explicit non-goal here.

Test plan

Added to tests/models/test_any_llm_model.py. Both new tests build a genuine any_llm.types.responses.ResponsesParams, so they assert against any-llm's real validator rather than a local stub, and they importorskip("any_llm") in line with the neighbouring tests.

  • test_any_llm_responses_path_sends_reasoning_as_a_mapping[False|True] — parametrized over non-streaming and streaming, since get_response and stream_response share _fetch_responses_response. Asserts params.reasoning == {"effort": "low", "summary": "concise"}.
  • test_any_llm_responses_path_omits_reasoning_when_unset — boundary: reasoning=None still sends None. This one passes on main; it guards against the fix regressing the unset case.

Pre-fix baseline on e943deda (fix reverted, tests kept):

$ uv run pytest tests/models/test_any_llm_model.py -q -k "reasoning_as_a_mapping or omits_reasoning_when_unset"
FAILED ...::test_any_llm_responses_path_sends_reasoning_as_a_mapping[False] - ValidationError: reasoning Input should be a valid dictionary
FAILED ...::test_any_llm_responses_path_sends_reasoning_as_a_mapping[True]  - ValidationError: reasoning Input should be a valid dictionary
2 failed, 1 passed, 36 deselected

Post-fix:

$ uv run pytest tests/models/test_any_llm_model.py -q
39 passed

Also run 3x with -W error::RuntimeWarning (39 passed each time) to confirm the streamed case leaves no unclosed async generator; the streaming test explicitly aclose()s the returned stream.

Full stack per AGENTS.md (.agents/skills/code-change-verification/scripts/run.sh):

$ env UV_DEFAULT_INDEX=https://pypi.org/simple bash .agents/skills/code-change-verification/scripts/run.sh
make format     -> 844 files left unchanged; ruff check --fix: All checks passed!
make lint       -> passed in 0s
make tests      -> passed in 34s
make typecheck  -> passed in 74s
code-change-verification: all commands passed.

$ make tests   # re-run to capture counts
6143 passed, 3 skipped, 2 warnings   (parallel)
45 passed, 4 skipped, 6146 deselected (serial)

git diff --check: clean.

Compatibility

No public API, signature, or field-order change. Only the AnyLLM Responses request payload is affected, and only for reasoning. Chat Completions, LiteLLM, and OpenAI Responses adapters are untouched. Reasoning() with every field unset now serializes to {} instead of raising — an empty mapping is valid for ResponsesParams.reasoning.

Related but distinct

#2823 / #2822 also stemmed from a pydantic model being iterable, but in _flatten_any_llm_reasoning_value — the response-side reasoning-text extraction on the chat-completions path. This is the request-side payload builder on the Responses path: different function, different direction, different failure mode. That fix is merged and unaffected here.

Issue number

No GitHub issue. Self-identified defect with the live repro above, following the precedent in #3700.

Checks

  • I've added new tests, if relevant
  • I've run .agents/skills/code-change-verification/scripts/run.sh
  • I've confirmed all verification steps pass
  • If using Codex, I've run /review before submitting this PR

_to_dump_compatible only materializes lazy iterables, and a pydantic model
iterates as key/value pairs, so ModelSettings.reasoning was serialized as a
list of pairs. any-llm types ResponsesParams.reasoning as a mapping, so every
AnyLLM Responses request configuring reasoning failed validation before
reaching the provider. Dump the model instead.
@seratch seratch added this to the 0.19.x milestone Aug 3, 2026
@seratch
seratch enabled auto-merge (squash) August 3, 2026 04:28
@seratch
seratch disabled auto-merge August 3, 2026 04:29
@seratch
seratch enabled auto-merge (squash) August 3, 2026 04:29
@seratch
seratch merged commit 0c3844a into openai:main Aug 3, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants