feat: Add content_document_file(), content_document_url(), and other file improvements - #345
Merged
Conversation
All three major providers accept plain text, Markdown, CSV, and code files as first-class document input, and OpenAI additionally parses docx/xlsx. chatlas had no way to express this: users had to open(path).read() and string-interpolate, losing the filename, Anthropic's citation eligibility, and OpenAI's spreadsheet parsing. Add ContentDocument (mirroring ContentPDF's shape, plus a real mime_type since documents span many formats) and content_document_file(). Separately, content_pdf_url() eagerly downloaded a PDF's bytes even though Anthropic supports a `url` source and OpenAI's Responses API supports `file_url` directly -- burning bandwidth and inflating payloads against Anthropic's 32MB and OpenAI's 50MB request limits. ContentPDF.data is now Optional[bytes] (None when only a `url` is set), with a validator requiring at least one of data/url. A new _content_file.py holds the shared "resolve bytes, downloading and caching them back onto the content only if actually needed" helper (ensure_bytes), used by both ContentPDF and ContentDocument wherever a provider can't take a URL directly.
…y provider ChatOpenAI() (Responses) and ChatAnthropic() prefer a ContentPDF's/ ContentDocument's `url` when present -- the Responses API accepts `file_url` for any file type, and Anthropic accepts a `url` document source for PDFs, so neither needs to download the file. Documents on Anthropic are coerced to a `text/plain` source (decoded UTF-8), with a clear error for binary formats it can't extract text from (docx/xlsx) and for content that doesn't decode as UTF-8, instead of a raw UnicodeDecodeError. ChatOpenAICompletions() and ChatGoogle() have no generic URL-fetch mechanism, so they always resolve to real bytes via ensure_bytes() (downloading and caching once, if only a `url` was set). ChatGoogle() additionally rejects docx/xlsx, which Gemini can't extract text from. Also rejects image/heic and image/heif on every provider except ChatGoogle() (the only one that supports them), via a shared check_image_content_type_supported() helper. Touches _content_expand.py (tool-result expansion) and _inspect.py (the InspectAI bridge) as fallout from ContentPDF.data becoming Optional[bytes]: both read `.data` directly and needed to route through ensure_bytes()/the new Optional-aware branching instead.
Gemini accepts image/heic and image/heif, but the shared ImageContentTypes literal excluded them, so content_image_file() and content_image_url() had no way to produce them. Both now recognize the .heic/.heif extensions and data: URLs of those types; every provider except ChatGoogle() rejects them via check_image_content_type_supported() (added in the previous commit). Resizing a HEIC/HEIF image requires the optional pillow-heif package (Pillow can't open them on its own); without it, content_image_file() raises a clear ImportError naming the package and suggesting resize="none" instead of a confusing Pillow failure.
Update the multi-modal input section of chat.qmd for content_document_file(), the URL-passthrough behavior of content_pdf_url(), and the per-provider limits on docx/xlsx and heic/heif. Add the new function/type to the quartodoc reference nav (docs/_quarto.yml, regenerated docs/_sidebar.yml) and CHANGELOG entries under New Features.
This was referenced Jul 29, 2026
The existing guidance framed this purely as a size tradeoff. The constraints that actually matter are that uploads only work on three providers, and that a ContentUploaded is a provider-scoped id rather than data -- so a chat containing one can't be replayed elsewhere. Explain it once in the multi-modal input guide and cross-reference from content_document_file() and FileManager.upload().
content_document_file(), plus lazy PDF URLs and HEIC
Two spots asserted a narrowing the type system could have proven: Anthropic's media_type needed a cast (plus a comment explaining why it was safe) after check_image_content_type_supported(), and content_image_url() cast the result of a membership test. TypeGuard can't express either, since it only narrows the true branch -- an `if is_heic_heif(x): raise` guard needs the negative branch narrowed too. Add TypeIs and use it for both. check_image_content_type_supported() now returns the narrowed type rather than None, matching the raise-or-return shape of ensure_bytes() and leaving the OpenAI call sites untouched. It also takes ImageContentTypes rather than str, so it can no longer be handed an arbitrary string. Derive HEIC_HEIF_IMAGE_TYPES from its Literal via get_args so the tuple and the type can't drift, replacing a duplicate literal in _content_image.py. Pin the image media_type on Anthropic's wire format, which nothing covered.
…-input # Conflicts: # chatlas/_content.py
With stream=True the connection isn't returned to the pool until the body is consumed, and raise_for_status() never reads it -- so an HTTP error left the response open. Use requests.get() as a context manager. The temp file it streamed through was pointless: the function returns every byte in memory anyway, so writing to disk and reading back added a round-trip without lowering peak memory. Join the chunks directly. download_bytes() had no direct coverage (callers all mock it out), so add tests for the joined result and for closing on the error path. Reported by Copilot on #345.
Two gaps in the document content model: ContentDocument.url was unreachable from the public API. The field, the validator, and OpenAI's file_url passthrough branch all existed, but no constructor ever set it, so only a hand-built ContentDocument could use it. Add content_document_url(), mirroring content_pdf_url() -- including data: URL handling and the redirect-to-PDF guard. DOCUMENT_MIME_TYPES omitted .rtf/.doc/.odt/.xls, which OpenAI documents accepting. They fell through to the text/plain default, so Google would have received binary bytes labelled as text. Map them explicitly, and widen the Google rejection set (OFFICE_MIME_TYPES -> BINARY_DOCUMENT_MIME_TYPES) so they get the same convert-first error as docx/xlsx rather than being sent as a text Blob.
That a `*_url()` constructor doesn't eagerly download is the expected behavior, not something the guide needs to set up.
content_document_file(), plus lazy PDF URLs and HEICThe callout told readers to consult each provider's docs, and its example list went stale as soon as the binary office formats landed. A matrix answers the question directly. Keep the genuine caveat -- that chatlas sending a type doesn't mean every model behind that provider can interpret it -- as the callout.
content_document_file(), content_document_url(), and other file improvements
Revert the supported-content-types callout to its original two sentences, and cut the upload guidance back to prose: the heading, the windup, and the bulleted caveats were more scaffolding than the three facts warranted.
A *_url() constructor not eagerly downloading is expected behavior, not something the reference needs to set up. ContentPDF's data/url fields now match ContentDocument's terser wording; data still notes when it's None, since a nullable field has to.
`filename_from_url()` derives a filename from the URL path, but a `data:` URL has no path -- urlparse leaves `text/csv;base64,...` in `.path`, so `Path().name` split on the `/` and produced `csv;base64,YSxiCjEsMgo=` as the filename sent to the provider. `content_pdf_url()` already avoided this with `unique_pdf_name()`; the document path just didn't follow suit. Add the equivalent namer, keyed on the MIME type so the extension is still meaningful, and use it for the empty-path fallback too -- `"document"` hardcoded there collided across inputs, and the filename is how a model tells documents apart in a multi-file prompt.
An unbounded `requests.get()` hangs indefinitely on a stalled connection. That runs inline while building a request, so a silent socket makes the whole chat appear stuck with nothing to point at. `requests` applies the timeout per socket read rather than to the transfer as a whole, so this doesn't cap how long a large file may take -- only how long the connection may go silent.
`_typing_extensions.py` imports it directly, but it was only ever present transitively (openai pins `typing-extensions<5,>=4.11`). This branch widens the exposure: before `TypeIs`, Python 3.12 took every symbol from `typing` and needed the package not at all, where now it does. The floor is the release that added `TypeIs` (4.10), and the marker matches where the module actually falls back -- 3.13+ has everything in `typing`. Resolution is unaffected, since openai's floor is already higher.
The API treats `filename` and `file_url` as mutually exclusive and rejects requests carrying both: 400 Mutually exclusive parameters: 'input[1].content[0]'. Ensure you are only providing one of: 'file_id' or 'filename'. So every URL-only ContentPDF/ContentDocument on ChatOpenAI() failed -- the whole point of not downloading the bytes up front. The existing unit tests asserted this exact payload shape and passed, so they now also assert that `filename` is absent.
Coercing every text-ish document into Anthropic's `text/plain` source throws away the real MIME type, so without `title` the model gets no hint of what the attachment is or how to refer to it in a multi-document prompt. `filename` was already on hand and simply dropped.
content_pdf_url() named every file from a process-global counter, so `file_00N.pdf` depended on how many earlier calls happened in the process. That makes request bodies non-deterministic (it broke a body-matched VCR cassette purely on test ordering) and throws away a real name that's already in the URL. Now the last path segment wins when it ends in .pdf, with the counter as a fallback -- matching what content_document_url() already did. Reaching filename_from_url() from _content_pdf.py would have been circular (_content_document already imports parse_data_url from there), so both URL helpers move to _content_file.py, the module this branch added for logic shared by ContentPDF and ContentDocument. That also stops _content_image.py and _inspect.py from importing a URL helper out of the PDF module.
OpenAI's own endpoint only accepts `application/pdf` in `file.file_data` and 400s on anything else, verified live. But OpenAICompletionsProvider is the base for 14 OpenAI-compatible backends (Ollama, LMStudio, OpenRouter, Groq, ...) whose file support differs and changes independently of OpenAI's, so refusing non-PDF documents here would decide the question for all of them. Send what the caller asked for and let the configured backend answer. The unit test now pins the pass-through rather than a chatlas-side error, and notes why there's no cassette test to go with it.
Every provider test for ContentDocument was a pure serialization assertion against a fake provider, so nothing verified that the documented support matrix -- text/office documents per provider, PDF URL passthrough -- matched what these APIs actually accept. Recording these found two real bugs (the Responses API rejecting filename+file_url, and Chat Completions accepting only application/pdf), both of which the serialization tests had happily pinned. Adds two fixtures and three conftest helpers. Each asks a question that can only be answered from the attachment, so a provider that drops the document or receives bytes it can't parse fails instead of letting the model bluff from the prompt. offsite_memo.docx is a minimal hand-built OOXML package (validated against a real parser) rather than a python-docx dependency. No document counterpart for ChatOpenAICompletions: OpenAI's endpoint refuses them and the compatible backends that might not aren't recordable.
The CHANGELOG credited ChatOpenAICompletions() with binary office support it
doesn't have, and both it and the guide promised Anthropic citations. Documents
never set `citations: {"enabled": True}`, so no citation could ever come back --
three docstrings and error messages made the same promise about what
content_pdf_*() unlocks.
Replaces the prose with a per-provider list, each row now backed by a recorded
cassette or a live probe. Also documents the content_pdf_url() filename change.
cpsievert
commented
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You can now hand a text, Markdown, CSV, or code file to a chat the same way you already hand it an image or a PDF:
Previously the only way to do this was to inline the file yourself:
That works, but it throws away three things the providers give you for free when the file arrives as a document rather than as prompt text: the filename (so the model can refer to it, and so multi-file prompts stay distinguishable), eligibility for Anthropic's citations, and OpenAI's spreadsheet preprocessing — it parses up to the first 1,000 rows per sheet and attaches generated header/summary metadata instead of dumping raw cells into the context.
content_document_url()does the same for a remote file, without downloading it when the provider can fetch it itself.What each provider accepts
Document support varies more than image support does, so these helpers send what the provider can take and raise a specific, actionable error otherwise — rather than letting the API reject the request.
ChatOpenAI()ChatOpenAICompletions()ChatAnthropic()ChatGoogle().txt,.md, code filestext/plain).csv,.tsv,.json,.xml,.htmltext/plain).docx,.xlsx,.doc,.xls,.rtf,.odtimage/heic,image/heifThe last two rows are the other two changes in this PR:
PDF URLs are no longer downloaded when the provider can fetch them itself.
content_pdf_url()used to eagerly pull the bytes even though Anthropic acceptssource: {type: "url"}and the OpenAI Responses API acceptsfile_url. That spent bandwidth and pushed requests toward Anthropic's 32 MB and OpenAI's 50 MB payload ceilings for nothing. Providers that genuinely need bytes now fetch on first use and cache the result on the content object, so a file reused across turns downloads once. Resolves the long-standing TODO in_content_pdf.py.HEIC/HEIF images work on Gemini, which supports them natively — useful since it's what iPhones produce by default. Resizing them needs the optional
pillow-heifpackage, and its absence now says so instead of failing obscurely.Docs also now explain when to use
content_*_file()versuschat.files.upload(), which previously read as a pure size tradeoff. The constraints that actually matter: uploads work on only three providers, and aContentUploadedis a provider-scoped id rather than data, so a chat containing one can't be replayed elsewhere.Notes for review
ContentPDF.datais nowOptional[bytes]. A PDF may be a URL whose bytes haven't been fetched. Runtime behavior is unchanged and a validator enforces that at least one ofdata/urlis set, but downstream type checkers will see the change. Every read site goes through a sharedensure_bytes()helper that returns non-optionalbytesor raises naming the URL._content_expand.pyand_inspect.pyare touched only as fallout from this.Anthropic gets
text/plainregardless of the real MIME type. Its document block has exactly one text-ish source variant (PlainTextSourceParam, whosemedia_typeis literally"text/plain"), so any genuinely-decodable format is coerced to that on the way out whileContentDocument.mime_typekeeps the accurate value for providers that want it. Binary formats get a "convert this first" error instead of a rawUnicodeDecodeError.Why per-modality content classes instead of one generic
ContentFile. Anthropic'sDocumentBlockParam.sourceis a real union:PlainTextSourceParamwants decoded UTF-8,Base64PDFSourceParamwants base64, andURLPDFSourceParamis PDF-only with no text equivalent. Unifying would have meant a pile of mutually-exclusive nullable fields. ellmer keeps them separate too, though it has no document/audio/video content at all, so it hasn't had to answer this question.Narrowing is done with
TypeIs, notcast. Two places asserted a narrowing the type system could prove: Anthropic'smedia_typeandcontent_image_url()'s membership check. Note thatTypeGuardcannot express either, since it only narrows the true branch — anif is_heic_heif(x): raiseguard needs the negative branch narrowed, which is whatTypeIsadds.check_image_content_type_supported()returns the narrowed type rather thanNone, matching the raise-or-return shape ofensure_bytes().Binary formats are labelled explicitly.
.rtf/.doc/.odt/.xlswould otherwise fall through thetext/plaindefault and reach Gemini as binary bytes labelled as text. They now map to the MIME types Python's ownmimetypesreports, and the Google rejection set covers all six binary formats..ppt/.pptxare deliberately excluded — OpenAI's file-input docs don't list them.Verified:
pyrightclean,ruffclean, 226 tests passing across the content and provider-dispatch suites.ellmer parity
All three changes here are new capability, not ports — ellmer has none of them today:
content_image_*andcontent_pdf_*are the only file inputs;ContentTextis a plain string.content_pdf_url()downloads viahttr2::req_perform()and base64-encodes. Anthropic getssource: {type: "base64"}, OpenAI getsfile_data, Gemini getsinlineData— no provider ever sees a URL.If ellmer wants parity:
ContentDocument+content_document_file()/content_document_url(). An S7 class alongsideContentPDFwith amime_typeproperty, and the same per-provider dispatch inas_json():text/plaincoercion for Anthropic, real MIME type for OpenAI and Gemini, "convert first" errors for the binary office formats on Anthropic and Gemini.google_upload()'s existing MIME table (R/provider-google-upload.R:178-224) is a reasonable starting point for extension mapping.ContentPDFRemoteTODO already sitting inR/content-pdf.R:34-36— the same TODO chatlas inherited and this PR removes.content_pdf_url()keeps the URL, Anthropic and OpenAI Responses send it as-is, and Gemini/Completions fetch lazily. R's copy-on-modify semantics make the caching half harder than in Python: chatlas mutatescontent.datain place so a reused PDF downloads once, which in ellmer needs either an environment/R7 mutable field or a memoised fetch keyed on URL.image/heic/image/heifincontent_image_file()'s extension map (R/content-image.R:74-86), pass them through on Gemini, and error with a convert-to list elsewhere. This one is harder there than here:resizedefaults to"low"in ellmer (not"none"), and every value but"none"routes throughmagick::image_read()(R/content-image.R:62,93-95). So the default path needs ImageMagick built with an HEIC delegate — a system-library condition, not apip install pillow-heif— which makes the unhappy path both the common case and one a package can't fix for the user.No ellmer issues track any of this (searched document, text file, csv, pdf url, heic), so these would be new.
Landing order
Part of a three-PR set (documents, audio, video). All three touch
ContentTypeEnum,ContentUnion, andcreate_content(), so whichever lands first forces a trivial rebase of the others. This one should go first — it establishes the content-model shape and the sharedensure_bytes()helper the other two build alongside.