feat: let a tool result carry an image - #73
Merged
Conversation
Since @-mentions learned about images, the model could look at a screenshot the user attached but could not fetch one itself: `read_file` on a PNG answered "looks binary, use bash to inspect it", because a tool result was a String. The most obvious question about a picture — "is the button aligned in build/preview.png?" — needed a human to attach the file first. An attachment now travels as a sibling of the result it came back with, a block of its own in the same tool message carrying the call's id. The alternative was nesting it inside the tool_result, which is Anthropic's wire shape — but everything in smith that walks a message reads top-level blocks, and nesting would have made the session store, the token estimate, the compaction stages and both renderers recursive for one feature. Flat, they carry the new case unchanged; the one provider that wants it nested folds it back at serialization time, and a text-only result still serializes to the string it always did, so no running session loses its prompt cache to a reshaped prefix. `Tool#run` is untouched. A tool that can produce a picture says so by overriding `run_with_media`, which returns the text and the blocks together; the other sixteen leave it at its default and take the path they always took. State on the tool would have been the smaller diff and the wrong one — a ParallelTool is shared between fibers, so the blocks would belong to whichever call wrote last. `read_file` decides from the signature, like an @-mention does: a screenshot saved as `notes.txt` comes back as the PNG it is, a text file named `shot.png` is still read as text, and the ceiling is the same `[media] max_bytes`. How a file entered the context says nothing about what it costs once it is there. Its text-only entry point never encodes the bytes — recognising a format is twelve bytes of signature, base64 is a third of the file on top, and a caller with nowhere to put a block has no use for it. Providers that cannot take an image inside a tool result — the OpenAI shape has nowhere to put one, whatever it can do in a user message — get a line saying so, appended to the result's own text rather than added beside it: under role tool every adapter keeps only tool_result blocks, so a note next to the image would have vanished as quietly as the image did. That is the same reason compaction now writes its marker into the result instead of replacing the block with a text one, and why the desperate pass may drop what a tool returned but never what the user attached to the question being answered. MCP results stay text. The protocol has image content and the adapter flattens it in the protocol layer; opening that is its own change. Refs #64. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seit #31 kann das Modell einen Screenshot ansehen, den der Nutzer angehängt hat — aber sich keinen selbst holen:
read_fileauf ein PNG antwortete „looks binary, use bash to inspect it", weil ein Tool-Ergebnis einStringwar. Die naheliegendste Frage zu einem Bild — „sitzt der Button richtig inbuild/preview.png?" — brauchte einen Menschen, der die Datei vorher anhängt.Was sich ändert
Ein Anhang reist als Geschwister-Block der
Role::Tool-Nachricht, mit dertool_call_idseines Ergebnisses.Die Alternative wäre gewesen, ihn in den
tool_resultzu verschachteln — das ist Anthropics Drahtform. Aber alles in smith, das eine Nachricht durchläuft, liest Top-Level-Blöcke: Session-Store, Token-Schätzung, Kompaktierungsstufen und beide Renderer hätten für ein Feature rekursiv werden müssen. Flach tragen sie den neuen Fall unverändert; der eine Provider, der es verschachtelt haben will, faltet beim Serialisieren zusammen — und ein reines Text-Ergebnis serialisiert weiterhin zum selben String, sodass keine laufende Session ihren Prompt-Cache an ein umgeformtes Präfix verliert.Tool#runbleibt unangetastet. Ein Tool, das ein Bild liefern kann, sagt das durch Überschreiben vonrun_with_media; die anderen sechzehn lassen den Default stehen. Zustand am Tool wäre der kleinere Diff und der falsche gewesen — einParallelToolwird von mehreren Fibern geteilt.read_fileentscheidet nach der Signatur, wie eine@-Mention: derselbe Test, dieselbe Grenze[media] max_bytes. Der reine Text-Einstieg encodiert die Bytes nie — ein Format zu erkennen kostet zwölf Bytes, Base64 ein Drittel der Datei obendrauf.Provider, die kein Bild in einem Tool-Ergebnis nehmen können, bekommen eine Zeile, die das sagt — in den Ergebnistext gefaltet, nicht daneben: unter
role: "tool"behält jeder Adapter nurtool_result-Blöcke, ein Hinweis daneben wäre so still verschwunden wie das Bild. Aus demselben Grund schreibt die Kompaktierung ihren Marker jetzt in das Ergebnis, und der verzweifelte Durchgang darf verwerfen, was ein Tool geholt hat, nie aber das, was der Nutzer an die gerade beantwortete Frage gehängt hat.MCP-Ergebnisse bleiben textig — eine eigene Änderung.
Verifiziert
read_fileauf ein rotes 16×16-PNG → „Solid bright red."session.jsonundtranscript.jsonlder Session: null Treffer für das Base64, stattdessenmedia_refplus eine Datei untermedia/.crystal spec: 936 Beispiele, 0 Fehler (26 neu),crystal tool format --checksauber.Refs #64.
🤖 Generated with Claude Code