v0.4.8: media handling (core + plugins) & plugin fixes
What's Changed
- The spine is media. matbot could already let a tool show the model a file; now a person can attach one, and a tool holding a stored file can actually do the handing. One breaking change (
createAboutMatbotTooltakesservices), one new optional service (MediaStore, an alias ofFileStore), one new typed error, and no newMessageContentarm — the push path needed none.
Full Changelog: v0.4.7...v0.4.8
0.4.8 — a person can hand the model something to look at
Full detail is in CHANGELOG.md under 0.4.8; the reasoning lives in CLAUDE.md § Media and the new
docs/MEDIA.md.
Media, in three parts
User-supplied session media. Bytes arrive by value at the submission boundary and are gone from the
message before it is enqueued: open() writes each inline arm through the new MediaStore and replaces it
with a file-ref. What persists is always a reference, with no exception to police — store.set runs at
turn start and at every turn end, so a 5MB image inlined in the document would otherwise be ~6.7MB of
base64 riding two whole-document writes per turn for the rest of the session.
MediaStore is an alias of FileStore, so every existing implementation (filesystem, SQLite, OPFS,
Drive) is a candidate unchanged, and putting media on a different medium is a registration, not a port —
demonstrated live, with sessions on one backend and media on SQLite. UserContent is a deliberately narrow
subset of MessageContent, validated against a whitelist: a wire boundary must not accept a forged
tool-result, thinking block or marker into persisted history.
The pull path got its first producer. model-content was built, tested against a fake tool, and emitted
by nothing — so asked to examine a workspace PNG, a model would reason correctly that it needed the bytes
inline, find no tool that could do it, and fall back to bash, curl and PIL. workspace_action show is now
that producer. read structurally could not have been extended: a tool result is a value in the
transcript, so base64 there is 4/3 of the file persisted and re-sent every round, for something the
model cannot see.
Frontends. The web composer (paperclip, drag, paste, chips, restore-on-refusal, GET /media/<id>, media
inline in the bubble live and on reload) and Telegram (photos, documents, audio, voice, video, caption as
prose). single_turn gains attach?: string[] so a consulted model can be shown a stored file too.
Refusals happen at the boundary, naming the file
The alternative is a provider 400 part-way through a turn the user already believes was sent — and by then
the file-ref is in history, where it resolves into every subsequent outgoing copy and fails the session
for good. So MediaRejectedError carries a reason and the offending file, and nothing is enqueued on a
refusal. Three of the six reasons are about the bytes: bad base64, a magic-byte mismatch, and a type no
endpoint decodes — image/* being the one prefix that cannot be admitted by prefix alone, because a
provider tries to decode that arm where an unknown document or audio type degrades to a text note.
Two limits worth stating: the session total is derived by summing what the store holds rather than
counted, and the per-file cap is the 8MB residency budget rather than a second number. A file larger
than the outgoing-copy window can never be resident, so admitting one would only buy an attachment that is
stored, charged to the quota, and permanently invisible to the model with nothing having said so.
Also in this release
about_matbot reports the system prompt, attributed per plugin. SystemContextRegistry.parts(ctx)
returns each contribution with the name of the plugin that registered it, and build() now derives from it
— one traversal, so the text sent and the breakdown reported cannot drift. This is the breaking change:
createAboutMatbotTool needs the live machine to rebuild the prompt.
session_edit summarise — compact by meaning rather than by shape, rewriting a prefix as a two-part
hand-off document via one singleTurn. background at — run a prompt once, at a stated time, persisted
and cancellable like any schedule. ComposedCallContext.progress() — a tool_function body is a plain
async function and cannot yield, which left the progress event unreachable from the one place a long run
actually happens.
Notable fixes
- A mid-turn steer could interrupt a turn the user never saw. The decision tested
s.running, which is
true across the pump's whole queue — so it answers "something is running", never "still the one you
meant". The target turn'straceIdis now captured when the submission arrives and compared before
aborting. - Prior reasoning is replayed on a field, not as prose — it was leaking into visible answers.
parts()could pair a contribution with the wrong plugin, and drop one, corrupting the prompt
actually sent rather than merely the report.- The web client's media-URL cache never released a byte — and in the browser bundle those values are
blob:URLs, kept alive by being registered rather than referenced. It is now a byte-bounded LRU in the
transport that mints them. showrefused audio, which it advertises, becauseMIME_MAPhad no audio extension at all.background'ssleepleaked an abort listener per call, multiplied by the new chunked long-wait.- An attachment-only first turn gets a title — auto-titling read text blocks only.
Upgrading
createAboutMatbotTool(version) → createAboutMatbotTool(version, services). Nothing else in the public
surface changed incompatibly; MediaStore is optional and both hosts seed their own file area as the boot
default, so attachments work with no configuration. A deployment with no media store is completely
unaffected until someone attaches something, and is then told why rather than having it silently dropped.