feat(airt): capture generated media output in multimodal workflows#76
Merged
Conversation
The generated multimodal target now returns the model's full reply when it contains generated media (image/audio/video) instead of only the text content, so media-out targets — an image-out model, or a speech-to-speech target like Amazon Nova Sonic driven through a custom target — have their output image/ audio/video parts captured and rendered in findings/traces. Plain text replies still return `.content` (a str), so text-out targets are unchanged. Pairs with the SDK's media-out response capture + per-modality scoring. Bumps capability 1.5.0 -> 1.6.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to the SDK multimodal-output work (dreadnode-tiger #1891). Enables the
platform-generated multimodal workflow to capture what a target generates,
not just what it says.
@task targetnow returns the model's full reply when itcontains generated media (image / audio / video), so those output parts are
captured and rendered in findings & traces. Plain text replies still return
.content(a str), so text-out targets are unchanged.an image-out model, or a speech-to-speech target such as Amazon Nova Sonic
driven via a custom target (audio-in → audio-out).
1.5.0→1.6.0.Why this is safe
The change is gated on
content_partscontaining an image/audio/video kind.Text-only replies take the identical
reply.contentpath as before — no changeto the overwhelming majority (text-out) flow. Added a regression test asserting
both the media-kind guard and the text fallback.
Sequencing
Merge alongside dreadnode-tiger #1891 (SDK: media-out
response_to_parts+per-modality scoring). The SDK's
response_to_partsis what turns the returnedreply into typed output parts. Per-modality scoring of generated media is a
separate future step (the current
llm_judgestringifies its input, so it can'tyet judge a generated image/audio — that needs a multimodal judge scorer).
Test plan
pytest capabilities/ai-red-teaming/tests/test_attack_runner.py— all passpytest tests/test_capability_release_plan.py— passes with 1.6.0image/audio/video renders & plays in Overview + Traces