Skip to content

feat: Add content_document_file(), content_document_url(), and other file improvements - #345

Merged
cpsievert merged 25 commits into
mainfrom
feat/content-document-input
Jul 30, 2026
Merged

feat: Add content_document_file(), content_document_url(), and other file improvements#345
cpsievert merged 25 commits into
mainfrom
feat/content-document-input

Conversation

@cpsievert

@cpsievert cpsievert commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

You can now hand a text, Markdown, CSV, or code file to a chat the same way you already hand it an image or a PDF:

import chatlas as ctl

chat = ctl.ChatAnthropic()
chat.chat(
    ctl.content_document_file("q3-revenue.csv"),
    "Which region grew fastest?",
)

Previously the only way to do this was to inline the file yourself:

chat.chat(f"Which region grew fastest?\n\n{open('q3-revenue.csv').read()}")

That works, but it throws away three things the providers give you for free when the file arrives as a document rather than as prompt text: the filename (so the model can refer to it, and so multi-file prompts stay distinguishable), eligibility for Anthropic's citations, and OpenAI's spreadsheet preprocessing — it parses up to the first 1,000 rows per sheet and attaches generated header/summary metadata instead of dumping raw cells into the context.

content_document_url() does the same for a remote file, without downloading it when the provider can fetch it itself.

What each provider accepts

Document support varies more than image support does, so these helpers send what the provider can take and raise a specific, actionable error otherwise — rather than letting the API reject the request.

Input ChatOpenAI() ChatOpenAICompletions() ChatAnthropic() ChatGoogle()
.txt, .md, code files ✅ (sent as text/plain)
.csv, .tsv, .json, .xml, .html ✅ (sent as text/plain)
.docx, .xlsx, .doc, .xls, .rtf, .odt ❌ error: convert first ❌ error: convert first
Document from a URL ✅ URL passed through ⬇️ downloaded ⬇️ downloaded ⬇️ downloaded
PDF from a URL ✅ URL passed through ⬇️ downloaded ✅ URL passed through ⬇️ downloaded
image/heic, image/heif ❌ error: convert first ❌ error: convert first ❌ error: convert first

The last two rows are the other two changes in this PR:

PDF URLs are no longer downloaded when the provider can fetch them itself. content_pdf_url() used to eagerly pull the bytes even though Anthropic accepts source: {type: "url"} and the OpenAI Responses API accepts file_url. That spent bandwidth and pushed requests toward Anthropic's 32 MB and OpenAI's 50 MB payload ceilings for nothing. Providers that genuinely need bytes now fetch on first use and cache the result on the content object, so a file reused across turns downloads once. Resolves the long-standing TODO in _content_pdf.py.

HEIC/HEIF images work on Gemini, which supports them natively — useful since it's what iPhones produce by default. Resizing them needs the optional pillow-heif package, and its absence now says so instead of failing obscurely.

Docs also now explain when to use content_*_file() versus chat.files.upload(), which previously read as a pure size tradeoff. The constraints that actually matter: uploads work on only three providers, and a ContentUploaded is a provider-scoped id rather than data, so a chat containing one can't be replayed elsewhere.

Notes for review

ContentPDF.data is now Optional[bytes]. A PDF may be a URL whose bytes haven't been fetched. Runtime behavior is unchanged and a validator enforces that at least one of data/url is set, but downstream type checkers will see the change. Every read site goes through a shared ensure_bytes() helper that returns non-optional bytes or raises naming the URL. _content_expand.py and _inspect.py are touched only as fallout from this.

Anthropic gets text/plain regardless of the real MIME type. Its document block has exactly one text-ish source variant (PlainTextSourceParam, whose media_type is literally "text/plain"), so any genuinely-decodable format is coerced to that on the way out while ContentDocument.mime_type keeps the accurate value for providers that want it. Binary formats get a "convert this first" error instead of a raw UnicodeDecodeError.

Why per-modality content classes instead of one generic ContentFile. Anthropic's DocumentBlockParam.source is a real union: PlainTextSourceParam wants decoded UTF-8, Base64PDFSourceParam wants base64, and URLPDFSourceParam is PDF-only with no text equivalent. Unifying would have meant a pile of mutually-exclusive nullable fields. ellmer keeps them separate too, though it has no document/audio/video content at all, so it hasn't had to answer this question.

Narrowing is done with TypeIs, not cast. Two places asserted a narrowing the type system could prove: Anthropic's media_type and content_image_url()'s membership check. Note that TypeGuard cannot express either, since it only narrows the true branch — an if is_heic_heif(x): raise guard needs the negative branch narrowed, which is what TypeIs adds. check_image_content_type_supported() returns the narrowed type rather than None, matching the raise-or-return shape of ensure_bytes().

Binary formats are labelled explicitly. .rtf/.doc/.odt/.xls would otherwise fall through the text/plain default and reach Gemini as binary bytes labelled as text. They now map to the MIME types Python's own mimetypes reports, and the Google rejection set covers all six binary formats. .ppt/.pptx are deliberately excluded — OpenAI's file-input docs don't list them.

Verified: pyright clean, ruff clean, 226 tests passing across the content and provider-dispatch suites.

ellmer parity

All three changes here are new capability, not ports — ellmer has none of them today:

ellmer today
Text/CSV/code documents No constructor exists. content_image_* and content_pdf_* are the only file inputs; ContentText is a plain string.
PDF URL passthrough content_pdf_url() downloads via httr2::req_perform() and base64-encodes. Anthropic gets source: {type: "base64"}, OpenAI gets file_data, Gemini gets inlineData — no provider ever sees a URL.
HEIC/HEIF Not supported; docs say "PNG, JPEG, WebP, and non-animated GIF".

If ellmer wants parity:

  1. Add ContentDocument + content_document_file()/content_document_url(). An S7 class alongside ContentPDF with a mime_type property, and the same per-provider dispatch in as_json(): text/plain coercion for Anthropic, real MIME type for OpenAI and Gemini, "convert first" errors for the binary office formats on Anthropic and Gemini. google_upload()'s existing MIME table (R/provider-google-upload.R:178-224) is a reasonable starting point for extension mapping.
  2. Split PDF source from PDF bytes. This is the ContentPDFRemote TODO already sitting in R/content-pdf.R:34-36 — the same TODO chatlas inherited and this PR removes. content_pdf_url() keeps the URL, Anthropic and OpenAI Responses send it as-is, and Gemini/Completions fetch lazily. R's copy-on-modify semantics make the caching half harder than in Python: chatlas mutates content.data in place so a reused PDF downloads once, which in ellmer needs either an environment/R7 mutable field or a memoised fetch keyed on URL.
  3. Accept image/heic / image/heif in content_image_file()'s extension map (R/content-image.R:74-86), pass them through on Gemini, and error with a convert-to list elsewhere. This one is harder there than here: resize defaults to "low" in ellmer (not "none"), and every value but "none" routes through magick::image_read() (R/content-image.R:62, 93-95). So the default path needs ImageMagick built with an HEIC delegate — a system-library condition, not a pip install pillow-heif — which makes the unhappy path both the common case and one a package can't fix for the user.

No ellmer issues track any of this (searched document, text file, csv, pdf url, heic), so these would be new.

Landing order

Part of a three-PR set (documents, audio, video). All three touch ContentTypeEnum, ContentUnion, and create_content(), so whichever lands first forces a trivial rebase of the others. This one should go first — it establishes the content-model shape and the shared ensure_bytes() helper the other two build alongside.

All three major providers accept plain text, Markdown, CSV, and code
files as first-class document input, and OpenAI additionally parses
docx/xlsx. chatlas had no way to express this: users had to
open(path).read() and string-interpolate, losing the filename,
Anthropic's citation eligibility, and OpenAI's spreadsheet parsing.

Add ContentDocument (mirroring ContentPDF's shape, plus a real
mime_type since documents span many formats) and content_document_file().

Separately, content_pdf_url() eagerly downloaded a PDF's bytes even
though Anthropic supports a `url` source and OpenAI's Responses API
supports `file_url` directly -- burning bandwidth and inflating
payloads against Anthropic's 32MB and OpenAI's 50MB request limits.
ContentPDF.data is now Optional[bytes] (None when only a `url` is
set), with a validator requiring at least one of data/url.

A new _content_file.py holds the shared "resolve bytes, downloading
and caching them back onto the content only if actually needed"
helper (ensure_bytes), used by both ContentPDF and ContentDocument
wherever a provider can't take a URL directly.
…y provider

ChatOpenAI() (Responses) and ChatAnthropic() prefer a ContentPDF's/
ContentDocument's `url` when present -- the Responses API accepts
`file_url` for any file type, and Anthropic accepts a `url` document
source for PDFs, so neither needs to download the file. Documents on
Anthropic are coerced to a `text/plain` source (decoded UTF-8), with a
clear error for binary formats it can't extract text from (docx/xlsx)
and for content that doesn't decode as UTF-8, instead of a raw
UnicodeDecodeError.

ChatOpenAICompletions() and ChatGoogle() have no generic URL-fetch
mechanism, so they always resolve to real bytes via ensure_bytes()
(downloading and caching once, if only a `url` was set). ChatGoogle()
additionally rejects docx/xlsx, which Gemini can't extract text from.

Also rejects image/heic and image/heif on every provider except
ChatGoogle() (the only one that supports them), via a shared
check_image_content_type_supported() helper.

Touches _content_expand.py (tool-result expansion) and _inspect.py
(the InspectAI bridge) as fallout from ContentPDF.data becoming
Optional[bytes]: both read `.data` directly and needed to route
through ensure_bytes()/the new Optional-aware branching instead.
Gemini accepts image/heic and image/heif, but the shared
ImageContentTypes literal excluded them, so content_image_file() and
content_image_url() had no way to produce them. Both now recognize
the .heic/.heif extensions and data: URLs of those types; every
provider except ChatGoogle() rejects them via
check_image_content_type_supported() (added in the previous commit).

Resizing a HEIC/HEIF image requires the optional pillow-heif package
(Pillow can't open them on its own); without it, content_image_file()
raises a clear ImportError naming the package and suggesting
resize="none" instead of a confusing Pillow failure.
Update the multi-modal input section of chat.qmd for
content_document_file(), the URL-passthrough behavior of
content_pdf_url(), and the per-provider limits on docx/xlsx and
heic/heif. Add the new function/type to the quartodoc reference nav
(docs/_quarto.yml, regenerated docs/_sidebar.yml) and CHANGELOG
entries under New Features.

This comment was marked as resolved.

The existing guidance framed this purely as a size tradeoff. The
constraints that actually matter are that uploads only work on three
providers, and that a ContentUploaded is a provider-scoped id rather
than data -- so a chat containing one can't be replayed elsewhere.

Explain it once in the multi-modal input guide and cross-reference from
content_document_file() and FileManager.upload().
@cpsievert cpsievert changed the title feat: text/CSV/code document input, PDF URL passthrough, HEIC support feat: document input via content_document_file(), plus lazy PDF URLs and HEIC Jul 29, 2026
@cpsievert cpsievert changed the title feat: document input via content_document_file(), plus lazy PDF URLs and HEIC feat: Add content_document_file(), plus lazy PDF URLs and HEIC Jul 29, 2026
Two spots asserted a narrowing the type system could have proven:
Anthropic's media_type needed a cast (plus a comment explaining why it
was safe) after check_image_content_type_supported(), and
content_image_url() cast the result of a membership test.

TypeGuard can't express either, since it only narrows the true branch --
an `if is_heic_heif(x): raise` guard needs the negative branch narrowed
too. Add TypeIs and use it for both.

check_image_content_type_supported() now returns the narrowed type
rather than None, matching the raise-or-return shape of ensure_bytes()
and leaving the OpenAI call sites untouched. It also takes
ImageContentTypes rather than str, so it can no longer be handed an
arbitrary string.

Derive HEIC_HEIF_IMAGE_TYPES from its Literal via get_args so the tuple
and the type can't drift, replacing a duplicate literal in
_content_image.py. Pin the image media_type on Anthropic's wire format,
which nothing covered.
With stream=True the connection isn't returned to the pool until the
body is consumed, and raise_for_status() never reads it -- so an HTTP
error left the response open. Use requests.get() as a context manager.

The temp file it streamed through was pointless: the function returns
every byte in memory anyway, so writing to disk and reading back added
a round-trip without lowering peak memory. Join the chunks directly.

download_bytes() had no direct coverage (callers all mock it out), so
add tests for the joined result and for closing on the error path.

Reported by Copilot on #345.
Two gaps in the document content model:

ContentDocument.url was unreachable from the public API. The field, the
validator, and OpenAI's file_url passthrough branch all existed, but no
constructor ever set it, so only a hand-built ContentDocument could use
it. Add content_document_url(), mirroring content_pdf_url() -- including
data: URL handling and the redirect-to-PDF guard.

DOCUMENT_MIME_TYPES omitted .rtf/.doc/.odt/.xls, which OpenAI documents
accepting. They fell through to the text/plain default, so Google would
have received binary bytes labelled as text. Map them explicitly, and
widen the Google rejection set (OFFICE_MIME_TYPES ->
BINARY_DOCUMENT_MIME_TYPES) so they get the same convert-first error as
docx/xlsx rather than being sent as a text Blob.
That a `*_url()` constructor doesn't eagerly download is the expected
behavior, not something the guide needs to set up.
@cpsievert cpsievert changed the title feat: Add content_document_file(), plus lazy PDF URLs and HEIC feat: document input via content_document_file()/content_document_url(), plus lazy PDF URLs and HEIC Jul 29, 2026
The callout told readers to consult each provider's docs, and its
example list went stale as soon as the binary office formats landed. A
matrix answers the question directly. Keep the genuine caveat -- that
chatlas sending a type doesn't mean every model behind that provider can
interpret it -- as the callout.
@cpsievert cpsievert changed the title feat: document input via content_document_file()/content_document_url(), plus lazy PDF URLs and HEIC feat: Add content_document_file(), content_document_url(), and other file improvements Jul 29, 2026
Revert the supported-content-types callout to its original two
sentences, and cut the upload guidance back to prose: the heading,
the windup, and the bulleted caveats were more scaffolding than the
three facts warranted.

This comment was marked as resolved.

A *_url() constructor not eagerly downloading is expected behavior, not
something the reference needs to set up. ContentPDF's data/url fields now
match ContentDocument's terser wording; data still notes when it's None,
since a nullable field has to.
`filename_from_url()` derives a filename from the URL path, but a `data:`
URL has no path -- urlparse leaves `text/csv;base64,...` in `.path`, so
`Path().name` split on the `/` and produced `csv;base64,YSxiCjEsMgo=` as
the filename sent to the provider.

`content_pdf_url()` already avoided this with `unique_pdf_name()`; the
document path just didn't follow suit. Add the equivalent namer, keyed on
the MIME type so the extension is still meaningful, and use it for the
empty-path fallback too -- `"document"` hardcoded there collided across
inputs, and the filename is how a model tells documents apart in a
multi-file prompt.
An unbounded `requests.get()` hangs indefinitely on a stalled connection.
That runs inline while building a request, so a silent socket makes the
whole chat appear stuck with nothing to point at.

`requests` applies the timeout per socket read rather than to the transfer
as a whole, so this doesn't cap how long a large file may take -- only how
long the connection may go silent.
`_typing_extensions.py` imports it directly, but it was only ever present
transitively (openai pins `typing-extensions<5,>=4.11`). This branch widens
the exposure: before `TypeIs`, Python 3.12 took every symbol from `typing`
and needed the package not at all, where now it does.

The floor is the release that added `TypeIs` (4.10), and the marker matches
where the module actually falls back -- 3.13+ has everything in `typing`.
Resolution is unaffected, since openai's floor is already higher.
The API treats `filename` and `file_url` as mutually exclusive and rejects
requests carrying both:

  400 Mutually exclusive parameters: 'input[1].content[0]'.
  Ensure you are only providing one of: 'file_id' or 'filename'.

So every URL-only ContentPDF/ContentDocument on ChatOpenAI() failed -- the
whole point of not downloading the bytes up front. The existing unit tests
asserted this exact payload shape and passed, so they now also assert that
`filename` is absent.
Coercing every text-ish document into Anthropic's `text/plain` source throws
away the real MIME type, so without `title` the model gets no hint of what the
attachment is or how to refer to it in a multi-document prompt. `filename` was
already on hand and simply dropped.
content_pdf_url() named every file from a process-global counter, so
`file_00N.pdf` depended on how many earlier calls happened in the process.
That makes request bodies non-deterministic (it broke a body-matched VCR
cassette purely on test ordering) and throws away a real name that's already
in the URL. Now the last path segment wins when it ends in .pdf, with the
counter as a fallback -- matching what content_document_url() already did.

Reaching filename_from_url() from _content_pdf.py would have been circular
(_content_document already imports parse_data_url from there), so both URL
helpers move to _content_file.py, the module this branch added for logic
shared by ContentPDF and ContentDocument. That also stops _content_image.py
and _inspect.py from importing a URL helper out of the PDF module.
OpenAI's own endpoint only accepts `application/pdf` in `file.file_data` and
400s on anything else, verified live. But OpenAICompletionsProvider is the base
for 14 OpenAI-compatible backends (Ollama, LMStudio, OpenRouter, Groq, ...)
whose file support differs and changes independently of OpenAI's, so refusing
non-PDF documents here would decide the question for all of them. Send what
the caller asked for and let the configured backend answer.

The unit test now pins the pass-through rather than a chatlas-side error, and
notes why there's no cassette test to go with it.
Every provider test for ContentDocument was a pure serialization assertion
against a fake provider, so nothing verified that the documented support
matrix -- text/office documents per provider, PDF URL passthrough -- matched
what these APIs actually accept. Recording these found two real bugs (the
Responses API rejecting filename+file_url, and Chat Completions accepting only
application/pdf), both of which the serialization tests had happily pinned.

Adds two fixtures and three conftest helpers. Each asks a question that can
only be answered from the attachment, so a provider that drops the document or
receives bytes it can't parse fails instead of letting the model bluff from the
prompt. offsite_memo.docx is a minimal hand-built OOXML package (validated
against a real parser) rather than a python-docx dependency.

No document counterpart for ChatOpenAICompletions: OpenAI's endpoint refuses
them and the compatible backends that might not aren't recordable.
The CHANGELOG credited ChatOpenAICompletions() with binary office support it
doesn't have, and both it and the guide promised Anthropic citations. Documents
never set `citations: {"enabled": True}`, so no citation could ever come back --
three docstrings and error messages made the same promise about what
content_pdf_*() unlocks.

Replaces the prose with a per-provider list, each row now backed by a recorded
cassette or a live probe. Also documents the content_pdf_url() filename change.
Comment thread docs/get-started/chat.qmd
@cpsievert
cpsievert merged commit 3579831 into main Jul 30, 2026
8 checks passed
@cpsievert
cpsievert deleted the feat/content-document-input branch July 30, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants