.NET: Python: [Bug]: File Citation Annotations Lost in Assistants API Streaming

[djw-bsn]

Description

When using the Azure AI Agents (or OpenAI Assistants) API, file citation annotations from TextDeltaBlock are silently discarded during streaming. Raw placeholder strings such as 【4:0†source】 pass through to the AG-UI layer and are rendered verbatim in the frontend instead of being resolved to meaningful citation links or metadata.

Note: This issue is specific to the Python SDK. An equivalent bug in the .NET SDK has already been addressed in #4114.

Background

Azure AI Agents is built on the Assistants API (thread/run/message model). When a run completes, the Azure AI service returns message content as MessageTextContent objects, each of which carries a .text.annotations array containing FileCitationAnnotation or FilePathAnnotation objects. These annotations are the only way to resolve 【n:n†source】 placeholder strings to actual file names, URLs, or metadata.

The agent_framework correctly handles annotations in two other code paths:

• agent_framework/azure/_chat_client.py (Chat Completions path) — lines 320–322
• agent_framework/openai/_responses_client.py (Responses API path) — lines 1114–1158

However, the shared Assistants API streaming client does not.

Root Cause

File: agent_framework/openai/_assistants_client.py
Lines: 556–559

During streaming, text delta blocks are converted to Content objects as follows:

if isinstance(delta_block, TextDeltaBlock) and delta_block.text and delta_block.text.value:
    contents=[Content.from_text(delta_block.text.value)],

delta_block.text.annotations is never read. The Content object is created with annotations=None. All downstream consumers — including agent_framework_ag_ui/_run_common.py which emits TextMessageContentEvent — receive content with no annotation metadata, so the raw placeholder strings are emitted verbatim into the AG-UI event stream.

Contrast With Working Code Paths

openai/_responses_client.py lines 1109–1158 shows the correct pattern:

text_content = Content.from_text(
    text=message_content.text,
    raw_representation=...,
)
if message_content.annotations:
    text_content.annotations = []
    for annotation in message_content.annotations:
        # handles file_citation, file_path, url_citation, container_file_citation
        text_content.annotations.append(...)

The same pattern exists in azure/_chat_client.py lines 298–322. The assistants streaming path is the only one that omits it.

Impact

This affects all users of the Assistants API streaming path (both Azure AI Agents and OpenAI Assistants) who use file search or file attachments. It is not an Azure-specific issue. The symptom is placeholder strings like 【4:0†source】 appearing verbatim in any downstream UI consuming AG-UI events.

Reproduction

1. Configure an Azure AI Foundry Agent with a file search tool and one or more uploaded knowledge files. Also reproducible with SharePoint based agent in Foundry.
2. Issue a query that triggers a file search and causes the assistant to produce a response with file citations.
3. Observe that TextMessageContentEvent.delta (emitted by agent_framework_ag_ui/_run_common.py _emit_text() at line 154) contains raw 【n:n†source】 strings.
4. Confirm that delta_block.text.annotations on the originating TextDeltaBlock is non-empty.

Expected Behaviour

Annotations from TextDeltaBlock.text.annotations are mapped to agent_framework Annotation objects and attached to the resulting Content object, consistent with the behaviour of _responses_client.py and _chat_client.py. Downstream consumers (including the AG-UI adapter) can then resolve or render citation placeholders correctly.

Actual Behaviour

delta_block.text.annotations is ignored. Content.annotations is None. Placeholder strings pass through to TextMessageContentEvent.delta (via _run_common.py:_emit_text() line 154: delta=content.text) and are rendered verbatim in the frontend.

Suggested Fix

In agent_framework/openai/_assistants_client.py, after creating Content.from_text(delta_block.text.value), add annotation mapping consistent with _responses_client.py:

if isinstance(delta_block, TextDeltaBlock) and delta_block.text and delta_block.text.value:
    text_content = Content.from_text(delta_block.text.value)
    if delta_block.text.annotations:
        text_content.annotations = []
        for annotation in delta_block.text.annotations:
            if annotation.type == "file_citation":
                text_content.annotations.append(
                    Annotation(
                        type="file_citation",
                        file_id=annotation.file_citation.file_id,
                        start_index=annotation.start_index,
                        end_index=annotation.end_index,
                        text=annotation.text,
                    )
                )
            elif annotation.type == "file_path":
                text_content.annotations.append(
                    Annotation(
                        type="file_path",
                        file_id=annotation.file_path.file_id,
                        start_index=annotation.start_index,
                        end_index=annotation.end_index,
                        text=annotation.text,
                    )
                )
    contents = [text_content]

The exact Annotation constructor fields should be aligned with the pattern already established in _responses_client.py lines 1119–1158.

Code Sample

Error Messages / Stack Traces

Package Versions

agent-framework-core v1.0.0rc2 agent-framework-ag-ui v1.0.0b260225 agent-framework-azure-ai v1.0.0rc2

Python Version

Python 3.12

Additional Context

• This issue affects the Python SDK only. The equivalent .NET bug should be fixed in .NET: Fix Usage and Annotations not present in AG-UI events (#3752) #4114 and may serve as a reference implementation.
• The AG-UI adapter (agent_framework_ag_ui/_run_common.py) emitting TextMessageContentEvent does not perform any annotation resolution itself. A fix in the assistants client is the correct layer - resolving the problem at the AG-UI layer would require duplicating knowledge about annotation formats that already lives in the client layer.
• Even with this fix applied, streaming delta annotations may be partial or incomplete. The thread.message.completed event - which carries the full, resolved annotation array - is also entirely unhandled in _assistants_client.py. I will raise this as a separate bug: ( thread.message.completed Event Unhandled in Assistants API Streaming)