openai_supports_tool_choice_required=False is still setting tool_choice to required for Responses

[pamelafox]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

This is another issue with gpt-oss on NIM (vLLM) with Responses API. I'm getting an error that tool_choice must be 'auto' for Harmony Responses:

pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: openai/gpt-oss-20b, body: {'message': "Only 'auto' tool_choice is supported in response API with Harmony None", 'type': 'BadRequestError', 'param': None, 'code': 400}

I've tried setting openai_supports_tool_choice_required=False but that isn't changing the tool_choice in the request to 'auto' as I hoped. I wonder if maybe it's not implemented for Responses API?

Full code here:

import asyncio
import os
import random
import logging
from typing import Literal

from dotenv import load_dotenv
from openai import AsyncOpenAI
from pydantic import BaseModel
from pydantic_ai import Agent, RunContext
from pydantic_ai.models.openai import OpenAIResponsesModel
from pydantic_ai.profiles.openai import OpenAIModelProfile
from pydantic_ai.providers.openai import OpenAIProvider
from rich.logging import RichHandler

load_dotenv(override=True)
logging.basicConfig(level=logging.DEBUG, format="%(message)s", datefmt="[%X]", handlers=[RichHandler()])
logger = logging.getLogger(__name__)

client = AsyncOpenAI(
    base_url=os.environ["NIM_ENDPOINT"],
    api_key="none")

model = OpenAIResponsesModel(
    os.environ['NIM_MODEL'],
    provider=OpenAIProvider(openai_client=client),
    profile=OpenAIModelProfile(openai_responses_requires_function_call_status_none=True, openai_supports_tool_choice_required=False),
)

class Weather(BaseModel):
    city: str
    temperature: int
    wind_speed: int
    rain_chance: int


class TriageResult(BaseModel):
    language: Literal["spanish", "english"]
    reason: str


async def get_weather(ctx: RunContext[None], city: str) -> Weather:
    """Returns weather data for the given city."""
    temp = random.randint(50, 90)
    wind_speed = random.randint(5, 20)
    rain_chance = random.randint(0, 100)
    return Weather(city=city, temperature=temp, wind_speed=wind_speed, rain_chance=rain_chance)


spanish_weather_agent = Agent(
    model,
    tools=[get_weather],
    system_prompt=("Eres un agente del clima. Solo respondes en español con información del tiempo para la ciudad pedida. " "Usa la herramienta 'get_weather' para obtener datos. Devuelve una respuesta breve y clara."),
)

english_weather_agent = Agent(
    model,
    tools=[get_weather],
    system_prompt=("You are a weather agent. You only respond in English with weather info for the requested city. " "Use the 'get_weather' tool to fetch data. Keep responses concise."),
)


# Triage agent decides which language agent should handle the request
triage_agent = Agent(
    model,
    output_type=TriageResult,
    system_prompt=(
        "You are a triage agent. Determine whether the user's request is primarily in Spanish or English. "
        "Return language (either 'spanish' or 'english') and reason (a brief explanation of your choice) "
        "Only choose 'spanish' if the request is entirely in Spanish; otherwise choose 'english'."
    ),
)


async def main():
    user_input = "Hola, ¿cómo estás? ¿Puedes darme el clima para San Francisco CA?"
    triage = await triage_agent.run(user_input)
    triage_output = triage.output
    print("Triage output:", triage_output)
    if triage_output.language == "spanish":
        weather_result = await spanish_weather_agent.run(user_input)
    else:
        weather_result = await english_weather_agent.run(user_input)
    print(weather_result.output)


if __name__ == "__main__":
    asyncio.run(main())

You can replicate with the endpoint that I shared with @DouweM last week, and the model "openai/gpt-oss-20b".

Example Code

Python, Pydantic AI & LLM client version

pydantic-ai==1.6.0

[pamelafox]

I did find a tool_choice issue when researching here: #2799
But that does not seem to be the same issue, from my skim of the thread.

[pydanticai-bot]

PydanticAI Github Bot Found 1 issues similar to this one:

1. "Only 'auto' tool_choice is supported in response API with Harmony. when i build a agent with gpt-oss-120b, if I assign a 'output_type', the tool_choice will be set to 'required', contradicted to the requirement. #2783" (90% similar)

[pamelafox]

That is also a similar issue, but I am not technically using structured outputs, I am using tools.

Here's what's sent over the wire:

{'input': [{'role': 'system', 'content':                    
                    "You are a triage agent. Determine whether the user's request is primarily in Spanish or English. Return language (either 'spanish' or                     
                    'english') and reason (a brief explanation of your choice) Only choose 'spanish' if the request is entirely in Spanish; otherwise                          
                    choose 'english'."}, {'role': 'user', 'content': 'Hola, ¿cómo estás? ¿Puedes darme el clima para San Francisco CA?'}], 'model':                            
                    'openai/gpt-oss-20b', 'stream': False, 'tool_choice': 'required', 'tools': [{'name': 'final_result', 'parameters': {'properties':                          
                    {'language': {'enum': ['spanish', 'english'], 'type': 'string'}, 'reason': {'type': 'string'}}, 'required': ['language', 'reason'],                        
                    'title': 'TriageResult', 'type': 'object'}, 'type': 'function', 'description': 'The final response which ends this conversation',                          
                    'strict': False}]}```

[pamelafox]

Maybe it goes down your structured outputs code path due to the tool returning a Pydantic BaseModel?

[DouweM]

@pamelafox You have output_type=TriageResult, so you are using structured output :) If we always set tool_choice=auto, that will make the default Tool Output mode less reliable as we're no longer forcing the model to call the final_result tool, so the suggestion in that issue to use NativeOutput instead (assuming this model supports it) stands.

But I agree we should respect openai_supports_tool_choice_required in OpenAIResponsesModel as well, by modifying this:

pydantic-ai/pydantic_ai_slim/pydantic_ai/models/openai.py

Lines 1152 to 1157 in 5cf8802

	if not tools:
	tool_choice: Literal['none', 'required', 'auto'] | None = None
	elif not model_request_parameters.allow_text_output:
	tool_choice = 'required'
	else:
	tool_choice = 'auto'

To match this from OpenAIChatModel:

pydantic-ai/pydantic_ai_slim/pydantic_ai/models/openai.py

Lines 470 to 478 in c9e9d93

	if not tools:
	tool_choice: Literal['none', 'required', 'auto'] | None = None
	elif (
	not model_request_parameters.allow_text_output
	and OpenAIModelProfile.from_profile(self.profile).openai_supports_tool_choice_required
	):
	tool_choice = 'required'
	else:
	tool_choice = 'auto'

Note that openai_supports_tool_choice_required is already set to False for harmony_model_profile:

pydantic-ai/pydantic_ai_slim/pydantic_ai/profiles/harmony.py

Lines 7 to 15 in c9e9d93

	def harmony_model_profile(model_name: str) -> ModelProfile | None:
	"""The model profile for the OpenAI Harmony Response format.
	See <https://cookbook.openai.com/articles/openai-harmony> for more details.
	"""
	profile = openai_model_profile(model_name)
	return OpenAIModelProfile(
	openai_supports_tool_choice_required=False, ignore_streamed_leading_whitespace=True
	).update(profile)

But that's not currently being applied automatically as there's no VLLMProvider that'll detect the gpt-oss name and use that profile, as for example OllamaProvider does:

pydantic-ai/pydantic_ai_slim/pydantic_ai/providers/ollama.py

Lines 45 to 55 in c9e9d93

	def model_profile(self, model_name: str) -> ModelProfile | None:
	prefix_to_profile = {
	'llama': meta_model_profile,
	'gemma': google_model_profile,
	'qwen': qwen_model_profile,
	'qwq': qwen_model_profile,
	'deepseek': deepseek_model_profile,
	'mistral': mistral_model_profile,
	'command': cohere_model_profile,
	'gpt-oss': harmony_model_profile,
	}

So between this and openai_responses_requires_function_call_status_none, I think we need a new VLLMProvider that'll use harmony_model_profile and then always set openai_responses_requires_function_call_status_none=True, so that new users can just use that provider and won't have to figure out these 2 profile settings themselves.

Would you be up for filing a PR to make OpenAIResponseModel respect openai_supports_tool_choice_required, and if you happen to have the time, also one add a new VLLMProvider?

[pamelafox]

Sure I can work on the PR.

(And sorry I missed that I was indeed setting output_type in this example, made original version a while ago)