GPT-5/o4-mini empty summary responses and AG-UI integration

[Joakim-T]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

Struggling with Pydantic AI, GPT-5 and the AG-UI integration.

Perhaps a related issue: #2645

OpenAI Responses API using GPT-5, o4-mini etc. send empty reasoning summary every single time even with summary set to None. This according to OpenAI is just how it is..

Problem

Pydantic AI's AG-UI integration generates invalid ThinkingTextMessageContentEvent objects with empty delta strings when using OpenAI reasoning models, causing AG-UI protocol validation failures that crash the streaming response.

This currently makes it impossible to use any OpenAI reasoning model (GPT-5 model/o4-mini etc.) with Pydantic AI's AG-UI integration, unless i missed something during all my debugging.

Error Output

The streaming starts successfully but then crashes:

data: {"type":"RUN_STARTED","threadId":"test-thread-123","runId":"test-run-456"}
data: {"type":"THINKING_TEXT_MESSAGE_START"}
data: {"type":"RUN_ERROR","message":"1 validation error for ThinkingTextMessageContentEvent\n  Value error, Delta must not be an empty string [type=value_error, input_value={'type': <EventType.THINKING_TEXT_MESSAGE_CONTENT>, 'delta': ''}, input_type=dict]"}

Expected vs Actual Behavior

Expected: Thinking events should either contain non-empty delta strings or be filtered out entirely if empty.

Actual: PydanticAI generates ThinkingTextMessageContentEvent objects with delta: '' (empty string), which fails AG-UI protocol validation.

The issue appears to be in PydanticAI's AG-UI event generation logic where:

1. The reasoning model generates thinking content (working correctly)
2. PydanticAI processes this into AG-UI events (working correctly)
3. Some thinking chunks result in empty delta strings (bug here)
4. AG-UI protocol validation rejects empty deltas (working correctly)

Workarounds

Use a Non-reasoning model
Just swapping to GPT-4.1 with no other changes works straight away.

Set "summary": detailed
Works whenever the model occasionally feels like generating a summary.

Custom Event Filtering
Would require require bypassing handle_ag_ui_request entirely so not ideal.

Suggested fix

AG-UI event generation should filter out or skip ThinkingTextMessageContentEvent objects when the delta is empty, rather than generating invalid events that fail protocol validation.

Example Code

Python, Pydantic AI & LLM client version

- **PydanticAI Version**: `pydantic-ai-slim==0.7.6`
- **Model**: `gpt-5` (OpenAI Responses API)
- **GPT-5 settings**: "reasoning": {"effort": "low", "summary": None},
"text": {"verbosity": "low"}
I have tested with text={"format": {"type": "text"}} also.
- **Integration**: `handle_ag_ui_request()` with FastAPI
- **Provider**: Azure OpenAI with `OpenAIResponsesModel`
Not using AsyncAzureOpenAI since Azure launched the new v1 APIs.

[dorukgezici]

I experienced the same thing, and have been using gpt-5-mini with the chat completions API for this reason. And I am using the Open AI provider directly, so it's not about Azure

[DouweM]

@Joakim-T Good catch -- looks like we should wrap this in an if part.content check (and remove the comment):

pydantic-ai/pydantic_ai_slim/pydantic_ai/ag_ui.py

Lines 457 to 462 in 21c87a5

	# Always send the content even if it's empty, as it may be
	# used to indicate the start of thinking.
	yield ThinkingTextMessageContentEvent(
	type=EventType.THINKING_TEXT_MESSAGE_CONTENT,
	delta=part.content,
	)

Can you test if that fixes the issue?

[Joakim-T]

Running 0.8.0 now.

@DouweM I tested but it seems it goes a bit deeper. :)

I tried what you suggested but THINKING_TEXT_MESSAGE_START and THINKING_TEXT_MESSAGE_END is always generated causing this error:
Agent execution failed: AGUIError: Cannot send 'THINKING_TEXT_MESSAGE_START' event: A thinking step is not in progress. Create one with 'THINKING_START' first.

How i got it to work was by changing 453-465 from

elif isinstance(part, ThinkingPart):  # pragma: no branch
    yield ThinkingTextMessageStartEvent(
        type=EventType.THINKING_TEXT_MESSAGE_START,
    )
    # Always send the content even if it's empty, as it may be 
    # used to indicate the start of thinking. 
    yield ThinkingTextMessageContentEvent(
        type=EventType.THINKING_TEXT_MESSAGE_CONTENT,
        delta=part.content,
    )
    stream_ctx.part_end = ThinkingTextMessageEndEvent(
        type=EventType.THINKING_TEXT_MESSAGE_END,
    )

to

elif isinstance(part, ThinkingPart):  # pragma: no branch
    if part.content and part.content.strip():
        yield ThinkingTextMessageStartEvent(
            type=EventType.THINKING_TEXT_MESSAGE_START,
        )
        yield ThinkingTextMessageContentEvent(
            type=EventType.THINKING_TEXT_MESSAGE_CONTENT,
            delta=part.content,
        )
        stream_ctx.part_end = ThinkingTextMessageEndEvent(
            type=EventType.THINKING_TEXT_MESSAGE_END,
        )
    # If no content, don't generate any thinking events at all

and lines 480-485 from

elif isinstance(delta, ThinkingPartDelta):  # pragma: no branch
    if delta.content_delta:  # pragma: no branch
        yield ThinkingTextMessageContentEvent(
            type=EventType.THINKING_TEXT_MESSAGE_CONTENT,
            delta=delta.content_delta,
        )

to

elif isinstance(delta, ThinkingPartDelta):  # pragma: no branch
    if delta.content_delta and delta.content_delta.strip():  # pragma: no branch
        yield ThinkingTextMessageContentEvent(
            type=EventType.THINKING_TEXT_MESSAGE_CONTENT,
            delta=delta.content_delta,
        )

Now Responses API with GPT-5/o4-mini is happy again.
But perhaps this causes other issues based on the comment that was in the code?

 # Always send the content even if it's empty, as it may be 
 # used to indicate the start of thinking.

[DouweM]

Agent execution failed: AGUIError: Cannot send 'THINKING_TEXT_MESSAGE_START' event: A thinking step is not in progress. Create one with 'THINKING_START' first.

@Joakim-T That's interesting, we're not currently using THINKING_START at all. That must be new.

Are you getting that error on the ThinkingTextMessageEndEvent, even though the error says THINKING_TEXT_MESSAGE_START? Because adding if content: around ThinkingTextMessageContentEvent would result in ThinkingTextMessageStartEvent, ThinkingTextMessageEndEvent where we previously had ThinkingTextMessageStartEvent, ThinkingTextMessageContentEvent, ThinkingTextMessageEndEvent, so we were already sending it a ThinkingTextMessageStartEvent without THINKING_START, and what it really seems to be complaining about is that we're starting and ending a thinking text message without content in between...

How i got it to work was by changing 453-465 from ...

Yeah that wouldn't work correctly if an empty PartStartEvent(ThinkingPart(content='')) is followed by a PartDeltaEvent(ThinkingPartDelta(content_delta='non empty')), because that would yield a ThinkingTextMessageContentEvent without there having been a previous ThinkingTextMessageStartEvent... So we'd need to check in the ThinkingPartDelta branch whether the most recent event was a ThinkingTextMessageStartEvent, and if not, create one.

But perhaps this causes other issues based on the comment that was in the code?

@stevenh I suppose the comment was there exactly so we didn't have to do this? Looks like AG-UI used to allow empty ThinkingTextMessageContentEvent, but now does no longer.

[stevenh]

The logic on this was that if the model tells us its thinking but not what it's thinking about the UI should still be updated to reflect the thinking state, just without a message.

I checked the AG-UI code it's always required content for deltas, so it's just something I missed and only now causing an issue.

@Joakim-T what happens if you send the ThinkingTextMessageStartEvent without a ThinkingTextMessageContentEvent if the delta is empty? That would satisfy the goal to notify the client without breaking the message type validation.

[Joakim-T]

@DouweM @stevenh

I did some more tests and it gets complicated 😄

Original 0.8.0 error message

1 validation error for ThinkingTextMessageContentEvent
Value error, Delta must not be an empty string

DouweM proposed fix

elif isinstance(part, ThinkingPart):  # pragma: no branch
    yield ThinkingTextMessageStartEvent(
        type=EventType.THINKING_TEXT_MESSAGE_START,
    )
    if part.content:  # only wrap the content event
        yield ThinkingTextMessageContentEvent(
            type=EventType.THINKING_TEXT_MESSAGE_CONTENT,
            delta=part.content,
        )
    stream_ctx.part_end = ThinkingTextMessageEndEvent(
        type=EventType.THINKING_TEXT_MESSAGE_END,
    )

This does indeed partially work as backend now completes the whole run.

data: {"type":"RUN_STARTED","threadId":"test-thread-123","runId":"test-run-456"}
data: {"type":"THINKING_TEXT_MESSAGE_START"}
data: {"type":"THINKING_TEXT_MESSAGE_END"}
data: {"type":"TEXT_MESSAGE_START","messageId":"b11b8bbb-eddb-4fc1-b02e-272bcede96e1","role":"assistant"}
etc.

But in the frontend (@ag-ui/client 0.0.36 ) i then get:

AGUIError: Cannot send 'THINKING_TEXT_MESSAGE_START' event: A thinking step is not in progress. Create one with 'THINKING_START' first.

ag-ui/client expects THINKING_START and THINKING_END

Checked ag_ui.core and they are in there

uv run python -c "from ag_ui.core import EventType; print([attr for attr in dir(EventType) if 'THINKING' in attr.upper()])"

['THINKING_END', 'THINKING_START', 'THINKING_TEXT_MESSAGE_CONTENT', 'THINKING_TEXT_MESSAGE_END', 'THINKING_TEXT_MESSAGE_START']

But when i modify ag_ui.py to send the new correct sequence

data: {"type":"RUN_STARTED","threadId":"test-thread-123","runId":"test-run-456"}
data: {"type":"THINKING_START"}
data: {"type":"THINKING_TEXT_MESSAGE_START"}
data: {"type":"THINKING_TEXT_MESSAGE_END"}
data: {"type":"THINKING_END"}

I get a frontend error as it turns out THINKING_START THINKING_END is not in the ag-ui/clients 0.0.36 schema

"code": "invalid_union_discriminator",
"options": [
  "TEXT_MESSAGE_START",
  "TEXT_MESSAGE_CONTENT",
  "TEXT_MESSAGE_END",
  "TEXT_MESSAGE_CHUNK",
  "THINKING_TEXT_MESSAGE_START",
  "THINKING_TEXT_MESSAGE_CONTENT",
  "THINKING_TEXT_MESSAGE_END",
  "TOOL_CALL_START",
  "TOOL_CALL_ARGS",
  "TOOL_CALL_END",
  "TOOL_CALL_CHUNK",
  "TOOL_CALL_RESULT",
  "STATE_SNAPSHOT",
  "STATE_DELTA",
  "MESSAGES_SNAPSHOT",
  "RAW",
  "CUSTOM",
  "RUN_STARTED",
  "RUN_FINISHED",
  "RUN_ERROR",
  "STEP_STARTED",
  "STEP_FINISHED"

Long story short
I found this PR
ag-ui-protocol/ag-ui#301

So i guess the correct sequence (after they fix the client) will be either

THINKING_START
THINKING_TEXT_MESSAGE_START  
THINKING_TEXT_MESSAGE_CONTENT (delta="Let me think about this...")
THINKING_TEXT_MESSAGE_END
THINKING_END

or

THINKING_START
THINKING_TEXT_MESSAGE_START
(no THINKING_TEXT_MESSAGE_CONTENT - skipped due to empty content)
THINKING_TEXT_MESSAGE_END  
THINKING_END

?

But cant verify since client is broken.

So the current only reliable workaround i can think of is skipping all thinking in ag_ui.py since THINKING_START and THINKING_END seems to be a new thing?

It must be a nightmare for you guys to keep track of these dependencies.

[stevenh]

Yer they were added during the development of the AG-UI layer, but are not documented.

I've asked the copilotkit team to clarify the expected flow and will report back.

[stevenh]

The Copilotkit team have confirmed correct sequence should be:

Model sends details:

1. THINKING_START
2. THINKING_TEXT_MESSAGE_START
3. THINKING_TEXT_MESSAGE_CONTENT
4. THINKING_TEXT_MESSAGE_END
5. THINKING_END

No details:

1. THINKING_START
2. THINKING_END