feat(llmobs): add audio_parts to LLM messages#18622
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 4 commits intoJun 22, 2026
Merged
Conversation
Add an optional audio_parts field to LLMObs messages so audio can be collected on LLM spans. Introduces the AudioPart type ({mime_type, content, attachment_key}), a format_audio_part helper for provider integrations, and public LLMObs.annotate support for audio_parts on input/output messages.
The SDK emits inline base64 content; the backend offloads content larger than its inline threshold to an attachment. No provider integrations are wired up yet (follow-up work).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0b1fa0ce8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Codeowners resolved as |
…arts Address Codex review: the public Messages wrapper rejected an audio part with neither content nor attachment_key, but allowed both. The backend contract expects exactly one per part, so reject the both-present case as well.
|
ncybul
approved these changes
Jun 18, 2026
Collaborator
Author
|
/merge |
|
View all feedbacks in Devflow UI.
This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
The expected merge time in
|
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
llmobs-audio-message-parts
branch
June 22, 2026 20:46
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Jun 25, 2026
## Description Phase 1 of LLMObs audio support: wire the **OpenAI** integration to emit `audio_parts` on chat completions, building on the Phase 0 data model/helper (#18622). - **Input audio** — `input_audio` parts on `gpt-4o-audio-preview` chat messages are now captured as `audio_parts` (inline base64 `content` + `mime_type`) instead of being dropped behind an `[audio]` placeholder. The placeholder is kept in the readable text content for backwards-compatible rendering. - **Output audio** — assistant audio (`choice.message.audio.data`) is captured as `audio_parts`, and the audio transcript is surfaced as the output message `content` (the API returns `content: null` for audio responses). - Adds `audio_mime_type_from_format(...)` to map OpenAI audio `format` values (`wav`, `mp3`, `flac`, `opus`, `pcm16`, ...) to MIME types. - Refactors `_extract_content_parts` to return both the readable text and the collected `AudioPart`s. Scope note: audio-in-chat payloads are small (conversational utterances), so they sit comfortably under the 5 MB per-span event cap. Whisper STT, TTS, and the Realtime API are deferred to follow-up PRs — Whisper in particular needs a size-guard design since transcription uploads can reach 25 MB (>5 MB span cap). **Also includes a related streaming fix:** the streamed-output branch of `openai_set_meta_tags_from_chat` discarded its tool results and leaked a stale `extracted_tool_results` value from the input-message loop, which could attribute a streamed output's (ReAct) tool result to the preceding input message. Now each streamed message uses its own tool results; covered by a regression test. Stacked on `llmobs-audio-message-parts` (#18622); will retarget to `main` once that merges. <img width="1039" height="580" alt="Screenshot 2026-06-25 at 11 37 50 AM" src="https://github.com/user-attachments/assets/db50ddf3-e47d-41f6-8a80-cb1848b22371" /> [**Example trace**](https://ddstaging.datadoghq.com/llm/traces?query=%40ml_app%3Aaudio-chat-sdk-test%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ7_aiBBxX9nKgAAABhBWjdfYWlCQkFBQkFVN3p4aHN5YkFBQUEAAAAkMDE5ZWZmNmEtYzJiYi00MTVhLThiNWMtMDUwY2U3MTcxMGViAAAAbg%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=8755109387349262575&start=1782398105751&end=1782401705751&paused=false) ## Testing - Unit tests for `audio_mime_type_from_format` and `_extract_content_parts` (text/image/audio) in `tests/llmobs/test_integrations_utils.py`. - Updated the two existing multimodal chat tests to assert input `audio_parts`. - New end-to-end `test_chat_completion_audio_output` (hand-authored cassette) asserting output `audio_parts` + transcript-as-content, gated to `openai >= 1.45` (when `ChatCompletionAudio` was introduced). - Ran `llmobs` and `openai` (latest) suites locally via `scripts/run-tests`; all pass. ## Risks Low. Input/output text extraction is unchanged except for the added `audio_parts` field; the `[audio]`/`[image]` text markers are preserved. Output-audio handling only triggers when `choice.message.audio` is present. ## Additional Notes Claude session: `57bda7ea-70ec-482f-a5fe-973dbf97f8c6` Resume: `claude --resume 57bda7ea-70ec-482f-a5fe-973dbf97f8c6` ## Follow-ups (deferred, out of scope) Tracked from code review (Codex + subagent): - **Streamed audio output capture** — when `stream=True`, audio arrives via `delta.audio` chunks that `openai_construct_message_from_streamed_chunks` doesn't accumulate, so streamed audio responses aren't captured (also affects litellm, which reuses this path). Input audio and non-streamed output audio are captured. Planned to land with the realtime/streaming work. - **Streamed-branch `tool_results` parity** — the streamed-output branch attaches `tool_calls` but not `tool_results` (the non-streamed branch does). Pre-existing, unrelated to audio; deferred to avoid a behavior change (the `content=""` clear for streamed ReAct) without dedicated tests. Non-blocking notes: pcm16 audio is captured but not specially mapped (the UI can't render raw PCM); audio is emitted inline as base64 with no `attachment_key` offload (by design — backend offloads). Co-authored-by: zach.groves <zach.groves@datadoghq.com>
zurdooo
pushed a commit
that referenced
this pull request
Jun 30, 2026
## Description Phase 1 of LLMObs audio support: wire the **OpenAI** integration to emit `audio_parts` on chat completions, building on the Phase 0 data model/helper (#18622). - **Input audio** — `input_audio` parts on `gpt-4o-audio-preview` chat messages are now captured as `audio_parts` (inline base64 `content` + `mime_type`) instead of being dropped behind an `[audio]` placeholder. The placeholder is kept in the readable text content for backwards-compatible rendering. - **Output audio** — assistant audio (`choice.message.audio.data`) is captured as `audio_parts`, and the audio transcript is surfaced as the output message `content` (the API returns `content: null` for audio responses). - Adds `audio_mime_type_from_format(...)` to map OpenAI audio `format` values (`wav`, `mp3`, `flac`, `opus`, `pcm16`, ...) to MIME types. - Refactors `_extract_content_parts` to return both the readable text and the collected `AudioPart`s. Scope note: audio-in-chat payloads are small (conversational utterances), so they sit comfortably under the 5 MB per-span event cap. Whisper STT, TTS, and the Realtime API are deferred to follow-up PRs — Whisper in particular needs a size-guard design since transcription uploads can reach 25 MB (>5 MB span cap). **Also includes a related streaming fix:** the streamed-output branch of `openai_set_meta_tags_from_chat` discarded its tool results and leaked a stale `extracted_tool_results` value from the input-message loop, which could attribute a streamed output's (ReAct) tool result to the preceding input message. Now each streamed message uses its own tool results; covered by a regression test. Stacked on `llmobs-audio-message-parts` (#18622); will retarget to `main` once that merges. <img width="1039" height="580" alt="Screenshot 2026-06-25 at 11 37 50 AM" src="https://github.com/user-attachments/assets/db50ddf3-e47d-41f6-8a80-cb1848b22371" /> [**Example trace**](https://ddstaging.datadoghq.com/llm/traces?query=%40ml_app%3Aaudio-chat-sdk-test%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ7_aiBBxX9nKgAAABhBWjdfYWlCQkFBQkFVN3p4aHN5YkFBQUEAAAAkMDE5ZWZmNmEtYzJiYi00MTVhLThiNWMtMDUwY2U3MTcxMGViAAAAbg%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=8755109387349262575&start=1782398105751&end=1782401705751&paused=false) ## Testing - Unit tests for `audio_mime_type_from_format` and `_extract_content_parts` (text/image/audio) in `tests/llmobs/test_integrations_utils.py`. - Updated the two existing multimodal chat tests to assert input `audio_parts`. - New end-to-end `test_chat_completion_audio_output` (hand-authored cassette) asserting output `audio_parts` + transcript-as-content, gated to `openai >= 1.45` (when `ChatCompletionAudio` was introduced). - Ran `llmobs` and `openai` (latest) suites locally via `scripts/run-tests`; all pass. ## Risks Low. Input/output text extraction is unchanged except for the added `audio_parts` field; the `[audio]`/`[image]` text markers are preserved. Output-audio handling only triggers when `choice.message.audio` is present. ## Additional Notes Claude session: `57bda7ea-70ec-482f-a5fe-973dbf97f8c6` Resume: `claude --resume 57bda7ea-70ec-482f-a5fe-973dbf97f8c6` ## Follow-ups (deferred, out of scope) Tracked from code review (Codex + subagent): - **Streamed audio output capture** — when `stream=True`, audio arrives via `delta.audio` chunks that `openai_construct_message_from_streamed_chunks` doesn't accumulate, so streamed audio responses aren't captured (also affects litellm, which reuses this path). Input audio and non-streamed output audio are captured. Planned to land with the realtime/streaming work. - **Streamed-branch `tool_results` parity** — the streamed-output branch attaches `tool_calls` but not `tool_results` (the non-streamed branch does). Pre-existing, unrelated to audio; deferred to avoid a behavior change (the `content=""` clear for streamed ReAct) without dedicated tests. Non-blocking notes: pcm16 audio is captured but not specially mapped (the UI can't render raw PCM); audio is emitted inline as base64 with no `attachment_key` offload (by design — backend offloads). Co-authored-by: zach.groves <zach.groves@datadoghq.com>
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Jul 14, 2026
## Description Phase 0 of adding automatic image collection to LLM Observability, mirroring the merged `audio_parts` foundation (#18622). The dd-source backend carries a typed `Message.image_parts` field so images can ride at the wire level through the attachment pipeline; this PR adds the SDK-side foundation so an SDK can emit it. **No provider integrations are wired up yet** (that's follow-up work). - `ddtrace/llmobs/types.py`: new `ImagePart` TypedDict (`mime_type`, `content`, `attachment_key`) and `image_parts: list[ImagePart]` on `Message`. - `ddtrace/llmobs/_integrations/utils.py`: `format_image_part(data, mime_type)` — base64-encodes raw bytes, passes an already-encoded base64 string through. The shared primitive future provider integrations will use to attach images. - `ddtrace/llmobs/utils.py`: the public `Messages` wrapper now preserves and validates `image_parts` (each part needs `mime_type` and exactly one of `content` / `attachment_key`), so `LLMObs.annotate(...)` supports images on input/output messages. - `ddtrace/llmobs/_llmobs.py`: documents `image_parts` in the `annotate` message schema. Wire shape per message: `image_parts: [{mime_type, content}]` with inline base64 `content`. The SDK only emits inline `content`; the backend decides whether to keep it inline or offload it to an `attachment_key`. ## Testing New tests mirror the `audio_parts` suite: - `format_image_part` (bytes→base64, str passthrough) - `Messages` preserves/validates `image_parts` — inline-content, attachment-key, and error cases (not-a-list, not-a-dict, missing `mime_type`, neither, both, wrong types) - end-to-end: `LLMObs.annotate` with `image_parts` on input and output → emitted span event carries `meta.input/output.messages[].image_parts` Run with `scripts/run-tests --venv <llmobs py3.13> -- -- -k image`; `ruff format`/style clean on the changed files. ## Risks Low — additive. `image_parts` is a new optional message key, omitted when absent, so non-image messages are unchanged on the wire. The only behavior change is that the public `Messages` wrapper now preserves a key it previously dropped. Inline images count toward the existing 5 MB per-event size limit (an oversized event has its whole input/output replaced with the existing dropped-value placeholder); image-aware truncation is out of scope for this foundational PR. ## Additional Notes Phase 0; provider integrations (PydanticAI, then OpenAI / Anthropic / Bedrock / Google vision) are planned follow-ups. Mirrors the `audio_parts` foundation (#18622). Co-authored-by: jose.izaguirre <jose.izaguirre@datadoghq.com>
brettlangdon
pushed a commit
that referenced
this pull request
Jul 15, 2026
## Description Phase 0 of adding automatic image collection to LLM Observability, mirroring the merged `audio_parts` foundation (#18622). The dd-source backend carries a typed `Message.image_parts` field so images can ride at the wire level through the attachment pipeline; this PR adds the SDK-side foundation so an SDK can emit it. **No provider integrations are wired up yet** (that's follow-up work). - `ddtrace/llmobs/types.py`: new `ImagePart` TypedDict (`mime_type`, `content`, `attachment_key`) and `image_parts: list[ImagePart]` on `Message`. - `ddtrace/llmobs/_integrations/utils.py`: `format_image_part(data, mime_type)` — base64-encodes raw bytes, passes an already-encoded base64 string through. The shared primitive future provider integrations will use to attach images. - `ddtrace/llmobs/utils.py`: the public `Messages` wrapper now preserves and validates `image_parts` (each part needs `mime_type` and exactly one of `content` / `attachment_key`), so `LLMObs.annotate(...)` supports images on input/output messages. - `ddtrace/llmobs/_llmobs.py`: documents `image_parts` in the `annotate` message schema. Wire shape per message: `image_parts: [{mime_type, content}]` with inline base64 `content`. The SDK only emits inline `content`; the backend decides whether to keep it inline or offload it to an `attachment_key`. ## Testing New tests mirror the `audio_parts` suite: - `format_image_part` (bytes→base64, str passthrough) - `Messages` preserves/validates `image_parts` — inline-content, attachment-key, and error cases (not-a-list, not-a-dict, missing `mime_type`, neither, both, wrong types) - end-to-end: `LLMObs.annotate` with `image_parts` on input and output → emitted span event carries `meta.input/output.messages[].image_parts` Run with `scripts/run-tests --venv <llmobs py3.13> -- -- -k image`; `ruff format`/style clean on the changed files. ## Risks Low — additive. `image_parts` is a new optional message key, omitted when absent, so non-image messages are unchanged on the wire. The only behavior change is that the public `Messages` wrapper now preserves a key it previously dropped. Inline images count toward the existing 5 MB per-event size limit (an oversized event has its whole input/output replaced with the existing dropped-value placeholder); image-aware truncation is out of scope for this foundational PR. ## Additional Notes Phase 0; provider integrations (PydanticAI, then OpenAI / Anthropic / Bedrock / Google vision) are planned follow-ups. Mirrors the `audio_parts` foundation (#18622). Co-authored-by: jose.izaguirre <jose.izaguirre@datadoghq.com>
vlad-scherbich
pushed a commit
that referenced
this pull request
Jul 15, 2026
## Description Phase 0 of adding automatic image collection to LLM Observability, mirroring the merged `audio_parts` foundation (#18622). The dd-source backend carries a typed `Message.image_parts` field so images can ride at the wire level through the attachment pipeline; this PR adds the SDK-side foundation so an SDK can emit it. **No provider integrations are wired up yet** (that's follow-up work). - `ddtrace/llmobs/types.py`: new `ImagePart` TypedDict (`mime_type`, `content`, `attachment_key`) and `image_parts: list[ImagePart]` on `Message`. - `ddtrace/llmobs/_integrations/utils.py`: `format_image_part(data, mime_type)` — base64-encodes raw bytes, passes an already-encoded base64 string through. The shared primitive future provider integrations will use to attach images. - `ddtrace/llmobs/utils.py`: the public `Messages` wrapper now preserves and validates `image_parts` (each part needs `mime_type` and exactly one of `content` / `attachment_key`), so `LLMObs.annotate(...)` supports images on input/output messages. - `ddtrace/llmobs/_llmobs.py`: documents `image_parts` in the `annotate` message schema. Wire shape per message: `image_parts: [{mime_type, content}]` with inline base64 `content`. The SDK only emits inline `content`; the backend decides whether to keep it inline or offload it to an `attachment_key`. ## Testing New tests mirror the `audio_parts` suite: - `format_image_part` (bytes→base64, str passthrough) - `Messages` preserves/validates `image_parts` — inline-content, attachment-key, and error cases (not-a-list, not-a-dict, missing `mime_type`, neither, both, wrong types) - end-to-end: `LLMObs.annotate` with `image_parts` on input and output → emitted span event carries `meta.input/output.messages[].image_parts` Run with `scripts/run-tests --venv <llmobs py3.13> -- -- -k image`; `ruff format`/style clean on the changed files. ## Risks Low — additive. `image_parts` is a new optional message key, omitted when absent, so non-image messages are unchanged on the wire. The only behavior change is that the public `Messages` wrapper now preserves a key it previously dropped. Inline images count toward the existing 5 MB per-event size limit (an oversized event has its whole input/output replaced with the existing dropped-value placeholder); image-aware truncation is out of scope for this foundational PR. ## Additional Notes Phase 0; provider integrations (PydanticAI, then OpenAI / Anthropic / Bedrock / Google vision) are planned follow-ups. Mirrors the `audio_parts` foundation (#18622). Co-authored-by: jose.izaguirre <jose.izaguirre@datadoghq.com>
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Jul 18, 2026
## Description LLM Observability: auto-instruments the OpenAI **Realtime API** (`client.realtime.connect(...)`, sync and async), the only streaming-audio surface in the OpenAI SDK (chat-completions streaming carries no audio). The Realtime API is a bidirectional WebSocket event stream, so it can't reuse the request/response streaming path. Instead, this wraps the connection's `send`/`recv`/`close` at the class level (every typed sub-resource send funnels through `RealtimeConnection.send`, and iteration funnels through `recv`) and feeds each observed event into a `_RealtimeState` machine that produces: - a **workflow** span for the session (connect → close), tagged with model, voice, input/output audio formats, and modalities; and - an **llm** child span per response turn (`response.created` → `response.done`), capturing the user and assistant transcripts as message content, token usage, and audio. Audio is emitted as `audio_parts` only for renderable formats. The Realtime API's default raw PCM (`audio/pcm|pcmu|pcma`) is not renderable, so those turns keep the transcript as message content (the playable-audio path is generic and future-proof). A reusable 5 MB inline size guard (`format_audio_part_with_guard`) keeps audio under the per-span-event limit. Builds on the audio `audio_parts` foundation (#18622) and the audio-in-chat work (#18695). ## Testing - Unit tests for the new audio helpers (`realtime_audio_format_to_mime`, `is_renderable_audio_mime`, `concat_base64_audio`, `format_audio_part_with_guard`). - `_RealtimeState` unit tests driving full turns (PCM transcripts-only, renderable `audio_parts`, session metadata, failed-status error, idempotent close, audio-only fallback marker). - An integration test driving a real patched `RealtimeConnection` over a fake websocket, asserting the resulting session + per-response LLMObs spans. - `lint` (fmt/style/typing/spelling/suitespec/error-log) all pass. Parametric/system tests are not included here, consistent with the audio-feature PRs that preceded this (#18622, #18695). ## Risks Low. Realtime instrumentation is gated on LLMObs being enabled and is fully wrapped in defensive try/except so it can never break a user's connection. Class-method wrapping is unpatched cleanly in `unpatch()`. No changes to existing chat/responses/completions paths beyond a backward-compatible `activate` kwarg on `BaseLLMIntegration.trace` and a `_get_model_provider` refactor (covered by existing tests). ## Additional Notes Claude session: `57bda7ea-70ec-482f-a5fe-973dbf97f8c6` Resume: `claude --resume 57bda7ea-70ec-482f-a5fe-973dbf97f8c6` Co-authored-by: zach.groves <zach.groves@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Phase 0 of adding automatic audio collection to LLM Observability. The dd-source backend recently added a typed
Message.audio_partsfield (DataDog/dd-source#462726) so audio can ride at the wire level through the attachment pipeline, but no SDK emits it yet.This PR adds the SDK-side foundation — the data model, a reusable helper, and the serialization plumbing — so audio can be carried on LLMObs messages. No provider integrations are wired up yet (that's follow-up work).
ddtrace/llmobs/types.py: newAudioPartTypedDict (mime_type,content,attachment_key) andaudio_parts: list[AudioPart]onMessage.ddtrace/llmobs/_integrations/utils.py:format_audio_part(data, mime_type)helper — base64-encodes raw bytes, passes an already-encoded base64 string through. The shared primitive future provider integrations will use to attach audio.ddtrace/llmobs/utils.py: the publicMessageswrapper now preserves and validatesaudio_parts(each part needsmime_typeand exactly one ofcontent/attachment_key), soLLMObs.annotate(...)supports audio on input/output messages.ddtrace/llmobs/_llmobs.py: documentsaudio_partsin theannotatemessage schema.Wire shape per message:
audio_parts: [{mime_type, content}]with inline base64content. The SDK only emits inlinecontent; the backend decides whether to keep it inline or offload it to anattachment_key.Testing
Messagespreserves/validatesaudio_parts(inline-content, attachment-key, and error cases incl. both-set and neither-set), andformat_audio_part(bytes→base64, str passthrough).LLMObs.annotatewithaudio_partson input and output → the emitted span event JSON carriesmeta.input/output.messages[].audio_parts.scripts/run-tests --venv <llmobs py3.13> -- -- -k audio→ 6 passed.ruff format/style +mypypass on the changed files.audio_partson both input and output messages is accepted by the LLMObs intake and renders in LLM Observability via the default (APM) path.Risks
Low. The change is additive:
audio_partsis a new optional message key, omitted when absent, so non-audio messages are unchanged on the wire. The only behavior change is that the publicMessageswrapper now preserves a key it previously dropped.Inline audio counts toward the existing 5 MB per-event size limit; an event exceeding it has its whole input/output replaced with the existing dropped-value placeholder. Audio-aware truncation is out of scope for this foundational PR.
Additional Notes
Phase 0 of a larger effort; provider integrations (OpenAI Whisper/TTS/Realtime, Gemini Live, Bedrock Nova Sonic, ElevenLabs, Deepgram, etc.) are planned follow-ups.
Claude session:
58e878cc-e73f-444a-bcf0-b3e667a1fa31Resume:
claude --resume 58e878cc-e73f-444a-bcf0-b3e667a1fa31