Incompatibility with vLLM Responses API

[pamelafox]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

The vLLM implementation of the Responses API is stricter than other implementations currently, and it will produce an error if it doesn't receive all the fields it expects, like the "status": None field.

Here's complete working code with the low-level openai package:
https://github.com/Azure-Samples/nim-on-azure-serverless-gpus-demos/blob/main/examples/openai_functioncalling_loop.py#L122

In other frameworks, we have resolved this by adding "status": None to the function call response, like in this PR for microsoft agent-framework:
https://github.com/microsoft/agent-framework/pull/1509/files

I did ask the vLLM maintainers if they can fix the issue to be less strict, and they are working on it, but I don't know how long it will take for it to get fixed and be reflected in all the services that use vLLM.

I replicated the issue with a gpt-oss deployed via NVIDIA NIM (which wraps vLLM) with the following code. If you ping me on Slack/LinkedIn/Twitter, I can share the endpoint URL.

import asyncio
import logging
import os
import random

from dotenv import load_dotenv
from openai import AsyncOpenAI
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIResponsesModel
from pydantic_ai.providers.openai import OpenAIProvider
from rich.logging import RichHandler

logging.basicConfig(level=logging.DEBUG, format="%(message)s", datefmt="[%X]", handlers=[RichHandler()])
logger = logging.getLogger("weekend_planner")

load_dotenv(override=True)
client = AsyncOpenAI(
    base_url=os.environ["NIM_ENDPOINT"],
    api_key="none")

model = OpenAIResponsesModel(os.environ["NIM_MODEL"], provider=OpenAIProvider(openai_client=client))

def get_weather(city: str) -> dict:
    """Returns the weather for the given city."""
    logger.info(f"Getting weather for {city}")
    if random.random() < 0.05:
        return {
            "city": city,
            "temperature": 72,
            "description": "Sunny",
        }
    else:
        return {
            "city": city,
            "temperature": 60,
            "description": "Rainy",
        }

agent = Agent(
    model,
    system_prompt="You are a helpful weather assistant.",
    tools=[get_weather],
)


async def main():
    result = await agent.run("what's the weather in Seattle?")
    print(result.output)


if __name__ == "__main__":
    logger.setLevel(logging.INFO)
    asyncio.run(main())

Python, Pydantic AI & LLM client version

pydantic==2.11.10
pydantic-ai==1.0.8
pydantic-ai-slim==1.0.8
pydantic-evals==1.0.8
pydantic-graph==1.0.8
pydantic-settings==2.11.0
pydantic_core==2.33.2

[DouweM]

@pamelafox Thanks for the report Pamela! The SDK type doesn't let me set it to None, so I picked 'completed' instead: #3246. Can you see if that PR works for you?

[DouweM]

@pamelafox Note that this is another issue you may run into with non-OpenAI Responses APIs: #3191

[pamelafox]

Alas, it only works with status: 'None'

I checked out your branch with:

uv add "pydantic-ai @ git+https://github.com/pydantic/pydantic-ai.git" --branch responses-function-call-status

I got the same error. I then went into the code and changed 'status' to None, and my request went through and worked fully.

So maybe it's not possible to support vLLM due to the typing?

[pamelafox]

I believe this is the relevant vLLM issue/fix:
vllm-project/vllm#26706

It was merged yesterday (likely from the vLLM maintainer that I saw at Pytorch, who identified the culprit), so that is good news for folks using vLLM directly.
I think it will be a while until the fix makes it into my environment, since I'm using an NVIDIA NIM image built on top of vLLM.

So if a patch isn't possible on pydantic-ai, it's possible that developers can wait for vLLM to be more compatible.

[DouweM]

@pamelafox We could ignore the typing error and pass None anyway, but I'd worry about sending that unconditionally as other Responses APIs may reject it as it's not one of the allowed values (the docs say "status string Optional. The status of the item. One of in_progress, completed, or incomplete") and None is not technically the same as optional / omitting the key entirely.

We could assume APIs will treat None the same as a missing key, but that still makes me a bit uncomfortable. I've confirmed OpenAI Responses is fine with it, but I think we should stick to the spec by default.

One option would be to add a field to the OpenAIModelProfile (https://ai.pydantic.dev/models/openai/#model-profile) that you can then set yourself. Or we could add a vLLMProvider that would set it automatically.

[DouweM]

@pamelafox I've updated the PR to make it a model profile setting. Your example should now work with this change:

model = OpenAIResponsesModel(
    os.environ['NIM_MODEL'],
    provider=OpenAIProvider(openai_client=client),
    profile=OpenAIModelProfile(openai_responses_requires_function_call_status_none=True),
)

You can also implement a vLLMProvider that injects that automatically:

class vLLMProvider(OpenAIProvider):
    def model_profile(self, model_name: str) -> ModelProfile | None:
        profile = super().model_profile(model_name)
        return OpenAIModelProfile(openai_responses_requires_function_call_status_none=True).update(profile)


model = OpenAIResponsesModel(os.environ['NIM_MODEL'], provider=vLLMProvider(openai_client=client))

I didn't include vLLMProvider (or should it be VLLMProvider) in the PR yet, but that could be a follow-up.

[pamelafox]

That's working for me! Given that the vLLM change is merged, I'm not sure it'd make sense to have a vLLM provider, since it may depend on what version of vLLM folks are running - hopefully vLLM won't always require status:None. I checked with NVIDIA and they only upgrade vLLMs for the NIM images every few months. Other things built on top of vLLM may have different upgrade cycles.