You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I maintain atomic-agents, which builds on instructor. We just got a downstream PR (Eigenwise/atomic-agents#274) adding video input support, and the contributor had to fall back to raw {"type": "video_url", "video_url": {"url": ...}} dicts, because instructor has no Video class to reach for. More providers accept OpenAI-style video_url content parts these days (MiniMax-M3 in that PR, Qwen-VL via DashScope, etc.), so I figure this will keep coming up.
Raw dicts do work, since convert_contents passes them through untouched. But they lose everything that makes Image/Audio/PDF pleasant: typing, the from_url/from_path helpers, and provider-specific conversion living in one place.
Concrete scope I have in mind:
A Video class in instructor/processing/multimodal.py following the existing Image/Audio/PDF pattern
from_url(), from_path(), from_base64() constructors with MIME validation for common video formats
to_openai() emitting an OpenAI-compatible video_url content part; provider converters that don't support video raise a clear error
Optional params like fps/detail where providers accept them
Prior art: #1850 asked for this for Gemini, and #1851 attempted it but was closed unmerged.
Happy to contribute the PR if this direction sounds right.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
I maintain atomic-agents, which builds on instructor. We just got a downstream PR (Eigenwise/atomic-agents#274) adding video input support, and the contributor had to fall back to raw
{"type": "video_url", "video_url": {"url": ...}}dicts, because instructor has noVideoclass to reach for. More providers accept OpenAI-stylevideo_urlcontent parts these days (MiniMax-M3 in that PR, Qwen-VL via DashScope, etc.), so I figure this will keep coming up.Raw dicts do work, since
convert_contentspasses them through untouched. But they lose everything that makesImage/Audio/PDFpleasant: typing, thefrom_url/from_pathhelpers, and provider-specific conversion living in one place.Concrete scope I have in mind:
Videoclass ininstructor/processing/multimodal.pyfollowing the existingImage/Audio/PDFpatternfrom_url(),from_path(),from_base64()constructors with MIME validation for common video formatsto_openai()emitting an OpenAI-compatiblevideo_urlcontent part; provider converters that don't support video raise a clear errorfps/detailwhere providers accept themPrior art: #1850 asked for this for Gemini, and #1851 attempted it but was closed unmerged.
Happy to contribute the PR if this direction sounds right.
All reactions