Skip to content

feat: Add content_video_file() and content_video_youtube() for Gemini video input - #347

Open
cpsievert wants to merge 4 commits into
mainfrom
feat/video-youtube-input
Open

feat: Add content_video_file() and content_video_youtube() for Gemini video input#347
cpsievert wants to merge 4 commits into
mainfrom
feat/video-youtube-input

Conversation

@cpsievert

@cpsievert cpsievert commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

You can now point a chat at a public YouTube video and ask about it — no upload, no download, no local copy:

import chatlas as ctl

chat = ctl.ChatGoogle()
chat.chat(
    ctl.content_video_youtube("https://www.youtube.com/watch?v=9hE5-98ZeCg"),
    "What are the three main points of this talk?",
)

Gemini fetches the video server-side, accepts up to 10 per request on 2.5+ models, and currently charges nothing for it.

A small local clip works the same way, sent inline:

chat.chat(
    ctl.content_video_file("bug-repro.mp4"),
    "At what point in this screen recording does the UI freeze?",
)

Previously the only route to video was the Files API:

chat.chat(chat.files.upload("bug-repro.mp4"), "When does the UI freeze?")

That path is genuinely right for large files — it already waits for Gemini to finish processing before returning — but it's a two-step round trip for a ten-second clip, and it cannot express a YouTube URL at all: there's no file to upload. This mirrors how images already get both an inline and a Files-API path, and the docs now say which to reach for.

What each provider accepts

Gemini is the only major provider that accepts video, so the interesting column is the first one. The rest raise a clear NotImplementedError naming Gemini rather than falling through to a generic "unknown content type".

Input ChatGoogle() / ChatVertex() ChatOpenAI() ChatOpenAICompletions() ChatAnthropic()
Local clip: .mp4, .mpeg, .mov, .avi, .flv, .mpg, .webm, .wmv, .3gpp ✅ inline ❌ error ❌ error ❌ error
Public YouTube URL ✅ passed through, no upload ❌ error ❌ error ❌ error
Large local file ⬆️ use chat.files.upload()

Inline video is capped by the ~100 MB request ceiling; past that, chat.files.upload() remains the answer and the docstring points there.

One finding worth flagging

Gemini's documented approach for YouTube URLs doesn't work through the SDK constructor. Part.from_uri() falls back to mimetypes.guess_type() when mime_type is omitted, and that can't infer anything from a youtube.com/watch?v=... URL, so it raises ValueError. Gemini genuinely wants no MIME type here — it determines the format itself. The provider builds Part(file_data=FileData(file_uri=url)) directly instead, with the reasoning in a comment at the call site so it doesn't get "simplified" back later. Verified against the installed google-genai, not inferred from docs.

Notes for review

YouTube URLs are modeled as their own ContentVideoUrl, not ContentUploaded with a synthetic MIME type. Nobody uploaded a YouTube video: it doesn't expire in 48 hours, it isn't listable or deletable through chat.files, and inventing a MIME type for it would put a falsehood into serialized turn history. The class shape stays generic ({url: str}) in case Gemini widens this to other URIs.

No generic "remote video URL" helper, because that isn't a verified Gemini capability — only YouTube is special-cased in their API.

No VCR cassette for a live YouTube round trip, since recording one needs a real GOOGLE_API_KEY. The dispatch-level tests cover the serialization logic, including a regression guard asserting file_data.mime_type is None so the Part.from_uri() trap can't come back.

Verified after merging main: pyright clean, ruff clean, 198 tests passing across the content and provider-dispatch suites, of which 29 are video-specific.

ellmer parity

New capability, not a port — ellmer has no video content of any kind today:

ellmer today
Inline video No constructor exists. content_image_* and content_pdf_* are the only file inputs; the Content classes are Text, Image, ToolRequest, ToolResult, Json, Uploaded, Thinking, PDF.
YouTube URLs Not reachable at all. There is no content type that can carry a URI without a MIME type.
Gemini Files API google_upload() has a video MIME table (R/provider-google-upload.R:210-216 — mp4, avi, mkv, mov, wmv, webm). That's the analogue of chat.files.upload(), not inline video.

If ellmer wants parity:

  1. Add a ContentVideo parent with inline and URL variants, plus content_video_file() / content_video_youtube(). The Gemini as_json() methods are short: inlineData with base64 + MIME type for the inline case, and fileData with a bare file_uri for the YouTube case. R sidesteps the trap that bit here — there's no Part.from_uri() equivalent doing helpful MIME guessing, so hand-building the list is the only option anyway.
  2. Reject video on the other providers with an actionable message. ellmer's as_json() dispatch already errors on unhandled content, but the default message won't tell the user that Gemini is the one provider that would have worked.
  3. Extend google_upload()'s MIME table while you're there: it's missing 3gpp, flv, mpg/mpeg, and includes mkv, which Gemini's inline-video docs don't list.

No ellmer issue tracks any of this (searched video, youtube), so it would all be new.

Landing order

Part of a three-PR set (documents, audio, video). All three touch ContentTypeEnum, ContentUnion, and create_content(). #345 has since landed and this branch is merged up to current main, so the only remaining overlap is with #346 (audio) — whichever of the two goes second needs a trivial re-merge of those three lists.

Introduces ContentVideoInline and ContentVideoUrl (plus the shared
ContentVideo base and VideoContentTypes literal) to represent Gemini video
input: small inline clips, and public YouTube URLs referenced with no
upload and no MIME type. content_video_file() and content_video_youtube()
are the public constructors, exported from chatlas and chatlas.types
alongside the existing image/PDF helpers.
ChatGoogle()/ChatVertex() send ContentVideoInline as inline Blob data and
ContentVideoUrl as a bare Part(file_data=FileData(...)) -- deliberately not
Part.from_uri(), which falls back to mimetypes.guess_type() and raises for
a YouTube watch URL that has no file extension to guess from.

ChatOpenAI(), ChatAnthropic(), and the OpenAI-compatible Chat Completions
providers (Groq, Mistral, Ollama, etc.) now raise a clear NotImplementedError
for video content instead of falling through to a generic "unknown content
type" error, consistent with how remote images are already rejected on
Google.

Also updates docs/chat.qmd and the quartodoc reference with a video input
section, and adds a CHANGELOG entry.
# Conflicts:
#	chatlas/_provider_openai.py
#	chatlas/_provider_openai_completions.py
#	docs/_quarto.yml
#	tests/test_provider_anthropic.py
#	tests/test_provider_openai.py
@cpsievert cpsievert changed the title feat(google): video input and direct YouTube URL support for Gemini feat: Add content_video_file() and content_video_youtube() for Gemini video input Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant