Video input: the modality seam is extensible, but the current failure mode is a silent drop

[johnlam1968]

What I found

DSH keeps its own modality vocabulary, separate from pi's, and it is narrower in one place and wider in another. Both halves are worth a look.

Sources are @ d743267388 (0.2.1-alpha.2).

The seam is already there, and documented as extensible

// packages/llm/llm/src/types.ts:230
/** Merge-extensible provider model modality vocabulary. */
export interface ModelModalityMap {
  text: 'text'
  image: 'image'
}

It's an interface with a doc comment inviting extension, so video can be added by declaration merging without touching the harness. That's the right shape — no objection to it, only a note that nothing currently does.

The failure mode is a silent drop

This is the part I'd flag. userContent() in the pi-ai adapter ends in:

// packages/llm/llm-pi-ai/src/context.ts:67
      default:
        // Other merge-extensible blocks are not user-input vocabulary for pi-ai.
        break

So an unrecognised block is discarded and the request still succeeds. A user who attaches a video would get a confident, fluent, completely invented answer about a video the model never saw, with nothing in the transcript to indicate the media was dropped.

Every other harness I checked makes omission visible instead:

harness	behaviour on unsupported media
Pi	substitutes "(image omitted: model does not support images)" — per-model gate, both user messages and tool results
ZCode (zai-org/zcode)	"[Media omitted from provider request because the selected model does not support video input.]", plus omittedVideoCount observability counters
OpenClaw	omits, and says so in the returned text

A silent drop is the one option of the three I'd argue against, independent of video: it converts a capability gap into a correctness bug that no one can diagnose from the transcript.

The catalog gap lands on DSH too

dsh-llm-pi-ai reads pi's generated catalog, so MiniMax-M3.1-Flash-Preview doesn't exist in DSH either. That's a one-line generator fix in pi and I've filed it there: https://github.com/earendil-works/pi/issues — the model works, it's just not listed.

Evidence that the capability is real

Measured this week against MiniMax's Anthropic-compatible endpoint. Fixture: 4 s 320×240 mp4, blue background, FRAME {n} burned into the picture, so a correct answer cannot come from metadata.

content block	result
{"type":"video","source":{"type":"base64","media_type":"video/mp4","data":…}}	200 — "A blue screen rapidly displays incrementing yellow-and-white frame numbers from 01 to 85."
{"type":"image","source":{"type":"base64","media_type":"video/mp4","data":…}}	400 — image data url media type "video/mp4" not supported (2013)
type:"audio"	400 — unsupported content type 'audio' (2013)

Surveying sst/models.dev (models/<lab>/<model>.toml, [modalities] input):

lab	model	declared input
Google	Gemini 4-Argon; Gemini 3-Pro / 3.1-Pro / 3-Flash	text, image, **video**, audio, pdf
Xiaomi	MiMo-V2.6-Pro	text, image, audio, **video**
MiniMax	M3, M3.1-Flash-Preview	text, image, **video**
Zhipu	GLM-4.6V	text, image, **video**
Moonshot	Kimi K3	text, image, **video**
ByteDance	Seed 2.1 Pro	text, image, **video**
Anthropic	Claude Opus 5.5, Sonnet 5.5	text, image, pdf
OpenAI	GPT-4o, GPT-5.6, GPT-6.1	text, image, pdf

Video input is present at every Chinese frontier lab in that table and absent at both Western ones. For a harness from a Chinese lab, that split is the argument: the capability is on the model menu DSH's users actually choose from.

The reference implementation is small

zai-org/zcode (Apache-2.0, Z.ai's GLM harness) shipped native video by patching its vendored AI SDK rather than forking the protocol. Its entire wire layer:

patches/@ai-sdk__anthropic@3.0.81.patch          48 lines
patches/@ai-sdk__openai-compatible@2.0.60.patch  44 lines

One branch each, keyed on mediaType.startsWith("video/"):

{ type: "video", source: { type: "base64", media_type: part.mediaType, data: convertToBase64(part.data) } }
// and for OpenAI-compatible routes
{ type: "video_url", video_url: { url: `data:${part.mediaType};base64,${convertToBase64(part.data)}` } }

Everything else in their implementation is the visible-omission policy and observability counters quoted in the table above.

Two questions

1. Is video modality on the roadmap, or is the merge-extensible ModelModalityMap deliberately left to adapters? I'd rather not build something that duplicates a planned harness change.
2. The silent default: drop — is that a deliberate fail-open, or just where the port stopped? If fail-open is intended I'd still suggest making it visible, since the alternative is a hallucinated answer with a clean-looking transcript.

What I'd do instead, per CONTRIBUTING

I understand external PRs aren't being taken, so this is the ecosystem path: a plugin, published with the dsh-plugin topic, that adds video input against MiniMax / GLM / Kimi / MiMo routes and reuses the same visible-omission rule rather than the silent drop.

For the pi side, pi2dsh runs unmodified pi extensions as native DSH plugins, so the same payload-level work covers both hosts — but that's the more fragile half of the plan (version alignment across pi / pi2dsh / DSH is not yet verified) and I'd rather build the DSH plugin directly.

Happy to be told this is already planned and get out of the way. I'll share the plugin either way.

[argszero]

Thanks for the careful write-up — the per-lab modality table and the "it cannot come from metadata" fixture are the useful half of a report like this.

I can only answer from the source (I am not a maintainer), but that bounds the question fairly tightly, and one part of your plan runs into a wall worth knowing before you build.

1. What is extensible and what is not

ModelModalityMap (packages/llm/llm/src/types.ts) is indeed declaration-mergeable, and so is ContentBlockMap — a video block type can be added from a plugin without touching the harness.

The serialization half is not extensible, and that is the half that matters. For pi-ai the conversion is userContent() in packages/llm/llm-pi-ai/src/context.ts — a module-level function, not a method — called from PiAiAdapter.streamWithSnapshot (adapter.ts:370). The package's public surface (llm-pi-ai/src/index.ts) exports PiAiAdapter, PiAiAdapterOptions, Config, recordKeyFor, supportedProtocols, name, inject, apply — the conversion function is not among them. And you cannot take the route over wholesale: registerAdapter is all-or-nothing and throws DUPLICATE_ADAPTER if any named route already has an adapter (packages/llm/llm/src/index.ts:431).

⇒ A plugin cannot widen the content vocabulary of an existing pi-ai route. The route that is open is a new provider route with its own adapter (ctx.llm.registerAdapter(providers, adapter) with route keys nothing else serves) — that adapter can put {type:'video', source:{type:'base64', media_type:…}} on the wire itself. That is a legitimate and documented shape (it is how the built-in adapters are mounted), but it is a new route, not "add video to MiniMax".

2. The default: drop — where the port stopped, or deliberate?

Read with the sibling protocol, it is neither fail-open nor fail-closed: it is per-protocol. In @deepseek-ai/dsh-llm-deepseek, protocols/messages/serialize.ts throws UNSUPPORTED_CONTENT for a user/tool-result block it cannot represent, while protocols/chat-completions/serialize.ts:186-189 drops it with exactly the comment your quote echoes ("Other merge-extensible blocks are not DeepSeek user-input vocabulary"). Same durable log, opposite outcomes, decided by which protocol the route defaults to. So "silent drop" is a property of the pi-ai adapter and of the chat-completions family, not a harness-wide decision about unknown blocks.

3. The half of your plan that is plugin-reachable: making omission visible

This one works today, on any route, without touching an adapter. The llm/stream waterfall gives you:

• the request is read-only: loop-built requests arrive deep-frozen and next(otherOptions) does not replace anything — the waterfall closes over its own arguments, so a listener cannot rewrite options in place;
• but the replacement path is to re-dispatch: build a fresh request object (same identity guard, use a WeakSet so you do not recurse) and call ctx.llm.stream(replacement). A listener that detects a block the target route cannot represent, substitutes the explicit placeholder text in the copy, and re-dispatches gets the visible-omission policy your table argues for, on the routes you name;
• do not set the loop-request marker on the copy — the agent loop installs an invariant that asserts a marked request's messages equal session.deriveMessages(), so a marked copy collides with it; an unmarked new object is simply outside that invariant's jurisdiction.

Two smaller notes from the same seam if you also touch responses: the chunk stream is yours to replace (const stream = next(); return wrap(stream) — the loop consumes exactly that, and it feeds both the durable assistant message and the replay record), blocks are assembled in first-seen order so dropping a whole block needs no reindexing, and if you ever drop an assistant content block you must drop the entry at the same position in finish.replayState.blocks or the harness trades a loud provider 400 for a silent signature loss.

For prior art on the request side: @argszero/cordis-plugin-history-sanitizer (npm) does exactly this shape for a different "the adapter cannot represent this block" case — it strips blocks from the outgoing request and re-dispatches, with an observe-only mode.

4. Your two questions

Whether video is on the roadmap is for the maintainers, not for me. What the source does show is that default: break is a stopped port rather than a statement about video: the comment treats unknown blocks as "not pi-ai user-input vocabulary", and the same code path already handles a video-typed image inconsistently (your 400 from image with media_type video/mp4 is the provider refusing, not DSH choosing). If you build the plugin, the new-route shape above is the one that can actually carry video today; the visibility half applies either way and is worth shipping even if the route work is still open.

[JannetGen]

Could you share a bit more detail about the issue and the steps to reproduce it? Logs or a minimal example would also help narrow down whether it’s a configuration problem or a bug in the harness. That should make it easier for others to investigate and suggest a fix.

[johnlam1968]

Thanks — both replies are useful, and one of them changes the plan. Repro question first.

This is the fourth ask, and the walls are already mapped

Before my own answer: video has been requested in this repo four times —
#4725 (2026-08-27),
#6050 from a user whose use
cases are lecture recordings, meeting minutes and accessibility (2026-09-09), this one, and
#9729 (2026-10-15, i.e. the
day after this one), asking for
essentially the same VideoBlock this post proposed. Credit where it is due — @Start-Ten's thread
argues from a person rather than from a capability table, and it deserves the same weight.

More useful is that the layer underneath this one is already diagnosed, repeatedly and in
detail, and it changes what a fix has to touch:

• #5837 — llm-pi-ai custom
  provider entries silently resolve to text-only when the model is genuinely multimodal. It names
  minimaxai/minimax-m3 and minimax/minimax-m3:free among the affected routes.
• #5367 — the same gap on the
  llm-deepseek side, where the catalog schema defaults an image-capable model to text-only.
• #6049 — read_image
  refusing a vision-capable model for the same reason.
• #8870 — a patch-ready
  proposal to read input_modalities from gateway listings (posted three times as feat(llm-pi-ai): read input modalities from gateway model listings (patch ready on a fork branch) #8870/feat(llm-pi-ai): read input modalities from gateway model listings (patch ready on a fork branch) #8873/
  feat(llm-pi-ai): read input modalities from gateway model listings (patch ready on a fork branch) #8876).
• #6060 — the client-side
  twin: assistant block kinds cannot be registered, and other is a deliberate JSON catch-all.

So there are two walls, not one, and this post only covered the second: a video block does not
exist (#9729), and the declaration vocabulary would discard it if it did. #8870's own test
note says so in one clause — after mapping an OpenRouter listing, values outside the pi-ai
vocabulary (file/audio/video) are dropped. I verified that chain here too: DEFAULT_INPUT = ['text'] at packages/llm/llm-pi-ai/src/config.ts:80, and the admission gate at
packages/api/session-controller/src/commands.ts:341 rejecting when the resolved
inputModalities is defined and lacks image.

One nuance those reports leave out, because it explains why this is so confusing in the field: the
guard fires only when inputModalities is defined and missing the modality. undefined
passes. The failure is not "undeclared models are refused" — it is that the resolution chain turns
an undeclared model into ['text'], which is defined, and then refuses on that. The user sees
"this model does not support video" and believes the model.

On the repro

There is no error to reproduce, and that is the finding: it is not a configuration problem and
not a crash — the request succeeds. Two halves.

1. The drop, minimally. packages/llm/llm-pi-ai/src/context.ts:67, userContent(),
switches on block.type with cases for text and image only and ends:

default:
  // Other merge-extensible blocks are not user-input vocabulary for pi-ai.
  break

Anything else in a user message never reaches the wire and nothing is logged. So the repro is:
put a video block (or any merge-extensible block the adapter does not know) into a user
message, send it, and read the request body — it succeeds, and the block is gone. There are no
logs to attach precisely because nothing recorded the omission; the artifact is the absence. A
fix looks like the visible-omission rule below: substitute explicit placeholder text, and count
it.

2. Why the omission costs something, provider-side. Measured 2026-10-13 against
https://api.minimaxi.com/anthropic/v1/messages (credential resolved from a running DSH
process; value never printed). The evidence is a direct HTTP probe, not a harness — worth saying
plainly, since "the model reads video" and "a harness can deliver a video" are different claims
and only the first is established by this test.

ffmpeg -hide_banner -loglevel error -y \
  -f lavfi -i "color=c=blue:s=320x240:d=4,drawtext=text='FRAME %{n}':fontcolor=yellow:fontsize=28:x=20:y=100" \
  -f lavfi -i "sine=frequency=880:duration=4" \
  -c:v libx264 -pix_fmt yuv420p -c:a aac -shortest probe.mp4

content block sent	result
{"type":"video","source":{"type":"base64","media_type":"video/mp4","data":…}}	200 — "…incrementing yellow-and-white frame numbers from 01 to 85."
{"type":"image","source":{"type":"base64","media_type":"video/mp4","data":…}}	400 — image data url media type "video/mp4" not supported (2013)
{"type":"audio",…}	400 — unsupported content type 'audio' (2013)

The frame number is burned into the picture with drawtext, so a correct answer cannot come from
metadata. The second row is the one that matters for a harness: the same bytes sent as an image
block are refused, which is why any video path has to rewrite the block type rather than smuggle a
MIME through an image slot.

A correction to my own post

The comparison table above attributes this placeholder to OpenClaw:

[Media omitted from provider request because the selected model does not support video input.]

That string is ZCode's, not OpenClaw's — 0 code-search hits in openclaw/openclaw. It comes
from zai-org/zcode apps/zcode-cli/packages/contracts/src/model/media-policy.ts, templated over a
typed reason, and ZCode's claim to omittedVideoCount counters belongs there too. OpenClaw's own
visible string is different —
packages/ai/src/provider-transcript-transform.ts:12:

const VIDEO_OMISSION = "(video omitted: provider does not support video input)"

Both omit visibly, so the argument in my post stands unchanged; the citation was mine and it was
wrong. Correcting it here rather than quietly, since the whole point of the report is that a
silent error is worse than a loud one.

On the structural correction (argszero)

Accepted — it kills the shape I sketched. userContent() is module-private, and
packages/llm/llm-pi-ai/src/index.ts exports PiAiAdapter, PiAiAdapterOptions, Config,
recordKeyFor, supportedProtocols, name, inject, apply — not the converter.
registerAdapter refuses a route another adapter holds: prepareRoutes throws
DUPLICATE_ADAPTER (packages/llm/llm/src/index.ts:431). Both verified at d743267388. So a
plugin cannot widen an existing pi-ai route's content vocabulary, and the open route is a new
provider route with its own adapter. That is the plan now.

One further constraint pushing the same way, which I had not checked: DSH has no onPayload
equivalent. GenerateOptions has no provider-options passthrough, and PiAiAdapter hands
models.streamSimple only the profile options plus
temperature/maxTokens/sessionId/signal/headers (adapter.ts:402). The only
request-side extension seam in the LLM layer is @deepseek-ai/dsh-deepseek-llm-api-extensions,
which merges top-level fields into the DeepSeek Messages body — a base-field collision is a
REQUEST_EXTENSION error — on the DeepSeek route only, and cannot rewrite messages[].content.
The one request-adjacent seam that does take structured input is the provider registration on
dsh-session-llm-title (ctx.sessionTitle.register({…})), and it selects the model that produces a
title — not the request body. So on the DSH side the omission decision has to be made before the
request is built, not at the wire — which happens to be what MiniMax-Code does, and is why its
design transfers.

Where I have to disagree — §2, the file paths. At the revision this post cites
(d743267388, 0.2.1-alpha.2), @deepseek-ai/dsh-llm-deepseek has one serializer,
packages/llm/llm-deepseek/src/serialize.ts (168 lines), not protocols/messages/serialize.ts;
packages/ contains no protocols directory at all, grep -rn chat-completions packages/llm/llm-deepseek/src returns nothing, and upstream main today lists the same layout.
Both DeepSeek paths throw: assistant content at line 36, user/tool-result content at line 67,
both through unsupported() → UNSUPPORTED_CONTENT (line 11). And chat-completions is not a
selectable DeepSeek protocol — config.ts:207 rejects the field outright:

llm-deepseek: protocol is not configurable; remove it and use a Messages-compatible baseURL

So "same durable log, opposite outcomes, decided by which protocol the route defaults to" does not
hold as written. What does hold is sharper, and it is why I asked: the asymmetry is per adapter
family, not per protocol. llm-deepseek throws; llm-pi-ai drops — and it drops on every
protocol pi-ai serves, because userContent() assembles the pi-ai Context before pi-ai chooses
an api (anthropic-messages, openai-completions, the Responses protocols). Same block, same
route, one api choice later: still dropped. If you are reading a newer revision with a protocols/
layout, say which commit and I will re-read it — I cannot substantiate the one I cited.

§3 accepted, with two sharpenings. (a) next takes no arguments:
'llm/stream'(this, options, next: () => AsyncIterable<StreamChunk>) (llm/src/index.ts:75), and
streamWithRegistration closes over options in the arrow it passes to ctx.waterfall.
Re-dispatch is therefore not a workaround but the only expressible route — which is what your
prior art does: @argszero/cordis-plugin-history-sanitizer@0.1.1 (npm, MIT) re-dispatches as a new
object with prepend: true. (b) On not marking the copy: agreed, and the sharper reason is that
the marker has a live consumer, not only a test invariant — isAgentLoopRequest gates the
title service (packages/session/session-title/src/index.ts:547), so a marked re-dispatched copy
enters onMainRequest and can schedule a second title call for the same pending revision.

ReplayEnvelope.blocks confirmed verbatim: "one entry per emitted block in first-seen stream
order. When assembly drops a block it drops the entry at the same position; entries whose length
does not match the emitted block count discard the whole envelope" (llm/src/types.ts:430).

One thing neither reply raised, which decides whether this ships

The durable half. ContentBlockMap is merge-extensible, but the harness states the admission
rule for a new modality at docs/subsystems/llm-streaming.md:34:

A new modality belongs in the merge-extensible map only when its adapter, UI, compaction, and
durable replay paths honor it.

Concretely: the v4 session format namespaces an unknown content tag as plugin:<type>
(packages/session/session-format-v3-to-v4/src/content.ts:15,45), so a plugin's video block is a
plugin-namespaced tag that also needs a session projection for the Web UI and a compaction path
that preserves it. Otherwise the request half works and the log cannot reconstruct the attachment
after a reload — model-visible must equal logged. There is a working precedent for the shape:
fileHandleText already projects every durable file to visible handle text on every route
(packages/llm/llm/src/content.ts:151), and a video a route cannot carry should degrade to
exactly that rather than vanish.

Plan, updated

• A new route with its own adapter, not "add video to MiniMax" — the registry forbids taking
  over an existing one — emitting
  {"type":"video","source":{"type":"base64","media_type":"video/mp4","data":…}}, with the
  decision made at ingress where it is reachable.
• Visible omission with a typed reason, in the style of ZCode's
  UnsupportedModelInputMediaKind. On llm/stream for routes that cannot carry the block, and at
  ingress for the rest.
• Three reference implementations, now read in source rather than cited from a blog:
  zai-org/zcode dispatches on mediaType.startsWith("video/") alone with no capability gate —
  two patches, 44 + 48 lines, covering both the Anthropic-style {type:"video", source:{…}} and the
  OpenAI-compatible {type:"video_url", video_url:{url:"data:…"}} encodings;
  MiniMax-AI/minimax-code (MIT, open source, pi-based) rides video through pi-ai's existing
  ImageContent with a video/* MIME and rewrites the serialized block at the payload boundary,
  recursing into nested tool-result content, deciding omission at ingress with typed reasons, and
  never setting fps; openclaw qualifies per route and substitutes visible text otherwise.
  Between them these answer every mechanism question the plan had.
• Cost control the others skip: MiniMax's OpenAPI exposes MediaSource.fps in [0.2, 5]
  (default 1) and max_long_side_pixel. Sampling is server-side, so that dial is the only lever
  on the token/latency trade-off, and no harness surveyed sets it.
• On the Pi side: the ecosystem says no quickly and without discussion — 工作区目录缺少 WRITE_OWNER 时，grantWrite 报 Win32 5 且错误信息未指明原因 #8886 closed
  not_planned 9 seconds after opening, 有些会话会莫名奇妙从已有的工作区跳到未分组里 #7326 7 seconds, and no release since v0.80.7 adds
  video. But StreamOptions.onPayload exists and runs as the last stop before the wire in all
  nine pi drivers, which is exactly the seam the extension needs. @weijiafu14's example in Feature request: agent-emitted images inline in assistant messages (assistant-side rendering) #2995
  is also worth reading: it is a working pi2dsh integration carrying a native image block through
  DSH's attachment service, which is evidence the delivery path is real today.

Question 1 (roadmap) still stands for the maintainers — as do the threads above, several of which
have sat unanswered for six weeks. I will post the plugin with the dsh-plugin topic either way,
and I would rather be told it is already planned. If the two walls are better addressed separately,
I am happy to split this: the declaration chain in #5837 is not specific to video and does not need
me in it.

Everything above is measured and sourced in a public repository — the harness landscape, the
discussion map, and the correction to my own table. Please read it rather than take the numbers on
trust, and correct anything that does not reproduce:

https://github.com/johnlam1968/harness-video-llm

Two caveats I would rather state than be caught on. First, the evidence that the model reads
video is a direct HTTP probe, not a harness run — the harness half of the claim is not yet
demonstrated, and that gap is what the repository is about. Second, the durable-replay half is
weaker than the request half and deserves scepticism: DSH's compaction seam keeps only text blocks
(packages/compaction/compaction-basic/src/summarizer.ts:220) and its offload plugin handles only
block.type === 'image', so a video block would be dropped at the first compaction with no
placeholder and no warning. Whatever gets built, that is the part that has to be right — and it is
the part none of the four existing implementations emphasises.