v7.0.0-rig: the embedder reads text, images, video and sound

Part of release 2026.10.6 (rig 7.0.0, sdk 5.2.0, lloyal-agents 7.3.0, media 0.4.0, binding 0.4.0, ui 0.4.0, host 0.5.0).

Ships with lloyal-ai 1.21.0, whose templates pin it.

The embedder reads text, images, video and sound

An embedding input is text and attachment roots in order: an image, a recording, a video or a window of one, with the text around them, embedded as one vector. EmbeddingGemma 2 with its projector is the default embedding service, and an embedding block that names no model takes it. Every input kind agrees with llama.cpp's own embedding server to a cosine of 0.999 or better. measure prices an input before anything runs, and a part the model can't read, an input over the context or a task the model doesn't frame is refused before any work.

The caller names a task (search, question answering, fact checking, classification and the rest) and a query or document role, and the encoder writes the model's own framing for it from the catalog. Each catalog model binds at its own context: 8,192 for EmbeddingGemma 2, 2,048 for nomic. A context or image detail outside what the model was trained for is refused before anything is fetched.

Two retrieval primitives join rig: fuseRanks, reciprocal rank fusion, and VectorIndex, Matryoshka vectors searched coarse to fine.

The SDK owns an embedding input

buildEmbedDelta builds an embedding input from interleaved text and media, embedCells measures it, and embedDelta encodes it and reads the pooled vector through lloyal.node's encodeMultimodal. dispose() answers once the context is freed, and every owner waits for it: rig's services, the host, initAgents and Rerank. A reranker is released only once the scores already queued have landed.

Services own their contexts

The embedder and the transcriber each run on a context their scope owns, one unit at a time through agents' FIFO gate, now exported as makePermits. No unit starts once the owning scope has begun ending, and the context is freed only after the running unit settles. A reranker lives as long as the scope that bound it. The host frees every context it lets go through one operation, and holds a session's id until its context is freed.

Video

• A composed video keeps its soundtrack for any model that hears, transcription or an embedding model, cut to the least of what each admits. The soundtrack is cut into windows of its own length, 5 s unless the harness sets attachments.video.soundtrackWindowSeconds, not at the interval between frame marks: a 10-minute clip is cut into 120 windows, not 33.
• videoWindow is a span of a stored video, as a video of its own, built from the stored blobs with nothing copied.
• Frames keep fusing past 5 min 20 s.
• A clip longer than the host takes is refused before any byte is pushed; there is no limit unless a harness sets one.
• A clip is read through any decoder, the procedure written once against ClipDecoder.
• Video ingest and the attachment share are harness.yml keys within declared bounds. The video keys are set at boot and fixed for the run.

Citations and the player

• A bare attachment address in prose is a link. A model that cites a moment or a page by writing its address, rather than a markdown link, has still cited it. @lloyal-labs/ui/prose exports REMARK_PLUGINS, the one grammar a harness view's markdown is read with, and linksOf reads with it too.
• The browser's own player moves to another moment of the same file.
• A pending audio-output recovery belongs to the clip that failed, so a new clip starts clean.
• A lightbox's video can wait: Lightbox takes waiting, and while it's set the poster says why the video isn't playing yet.

Versions

rig 7.0.0, sdk 5.2.0, lloyal-agents 7.3.0, media 0.4.0, binding 0.4.0, ui 0.4.0 and host 0.5.0. relay 0.3.1, desktop 0.4.1 and dev-tools 0.6.1 take the new versions, unchanged otherwise; channel-verify is unchanged. sdk, host and rig require lloyal.node 4.2.0 or later, the first with encodeMultimodal and the promise-returning dispose(). The abilities admit rig 7, and documents media 0.4: corpus 3.0.2, documents 0.3.2, web 3.0.1 and wikipedia 3.0.1, through the signed ability channel.

Upgrading

• sdk: dispose() returns Promise<void>, resolved once the native state is freed and never rejected. Await it before anything that needs the memory back.
• media and binding: VideoCapabilities.transcription is now soundtrack, and prepareVideo's policy takes soundtrack in its place.
• media and binding: a video's sampling carries soundtrackWindowMs, and soundtrackWindowMs(sampling, limits) reads it rather than the interval between time marks.
• rig: the Embedder contract reads parts and gains audio; Reranker no longer has dispose(), since its scope ends it. Qwen3-Embedding leaves the catalog.

Released alongside liblloyal v3.2.1 and lloyal.node 4.2.0.

What's Changed

• feat: video as a framework feature, and multimodal embedding with EmbeddingGemma 2 by @lloyal-research in #153

Full Changelog: v6.1.1-rig...v7.0.0-rig

v6.1.1-rig: a commit that fails for room costs its batch, not the pool

A commit that fails for room costs its batch, not the pool

When the pool commits a batch of sampled tokens and the context has no room for it, the agents in that batch give way and the pool goes on: each settles failed with the reason commit_failed, beside tool_error, its branch is pruned on the next tick, and every other agent keeps working and reports. Before, the same failure closed the whole pool partial. useAgent returns the agent that gave way, as it does for a tool error. A fatal commit, one the engine cannot recover from, still closes the pool, and what the pool rethrows is the store's own error, never a summary built from it.

Only the refusals the code means are refusals

A spawn the pool refuses for room is a SpawnRefusal. A throw on the way to a seat, in pricing, in a suffix prefill or in a replay, is now the run's error and settles in the spawn ledger as spawn_error, and any sibling stood down by the same throw as withdrawn, so a programming error is never reported as a lack of room. A replay hands the ledger entry to its replacement only once the replacement is active.

A heal stands down on content that is gone, and nothing else

@lloyal-labs/media exports ContentLost, raised when an attachment's content is no longer in the store. A recovery that rebuilds an agent's lineage stands down on that error alone; any other error is the run's.

Also

• The corpus ability's read_file reads from the line asked for. With a start line and no end line, the default window now begins at the start line and clips at the end of the file; before, it was anchored to the top of the file, so a read past line 100 returned nothing and an inverted range. Overlapping reads subtract exactly what was already delivered. The ability ships through the signed channel as lloyal/corpus@3.0.1.

Versions

lloyal-agents 7.2.0, rig 6.1.1, media 0.3.1. sdk, ui, binding, desktop, host and relay are unchanged. Every package requires lloyal.node 4.1.0 or later; 4.1.1 carries liblloyal 3.1.1, where a change of sampling parameters keeps the seed the chain has. The corpus ability is 3.0.1.

Upgrading

• The settle-failure reason union gains commit_failed; code that narrows it exhaustively must handle it.
• The spawn ledger's outcomes gain spawn_error and withdrawn beside the refusal kinds.

Ships with lloyal-ai 1.20.0, alongside liblloyal v3.1.1 and lloyal.node 4.1.1.

What's Changed

• fix(agents): the commit batch gives way; only the refusals the code means are refusals by @lloyal-research in #151
• Fix corpus reads with an omitted end line by @lloyal-research in #150

Full Changelog: v6.1.0-rig...v6.1.1-rig

v6.1.0-rig: native video, an OCI content store, and a player that shows what the model saw

A content store that speaks OCI

Every attachment a harness takes now lives in an OCI Image Layout, addressed by digest, with a manifest the store composed itself. A video is one manifest: its original as a source layer, the frames the model saw as representations, the transcript as another, and a typed config describing the timeline. A manifest is one the store committed; a blob that merely parses as a manifest is never read as one.

The original travels over the OCI distribution API: a chunked, resumable blob push (POST, PATCH with Content-Range, GET to resume, a closing PUT with the digest), staged and verified before it is admitted. The official distribution conformance suite's blob group runs against the host in CI. By default the push admits only video originals, by container signature, and serves no blob's bytes by digest; a pushed original is read through the attachment that holds it. BlobUploads(…, { registry: true }) opens the full registry when a harness wants one.

Reads are the contract too. Digests are verified on write and on whole reads; byte ranges are served without rehashing. The representation and source routes, the blob GET and the desktop attachment: scheme answer Range with 206, so a two-hour recording seeks like a local file. Writes require a matching Origin.

New in @lloyal-labs/media: materialize returns typed inputs (PreparedContent.inputs: each image as its admitted bytes, each video as one input of its frames on their timeline), and the store gains size, readRange, readWhole, stage() and pushBlob.

Video, end to end

The page decodes the clip with WebCodecs (captureFrames), pushes the original, and sends one references-only command, video:compose. The host validates the references, transcribes the soundtrack through the resident transcription service in windows aligned to the frame timestamps, and composes the manifest. The model then receives the frames as a video (MultimodalInput gains a video member), fused in pairs by Qwen-VL projectors, with the transcript beside the timestamps.

@lloyal-labs/ui ships a themable Video.js player. Its time slider shows the frame the model was given at every position, the transcript plays as captions, and a cited moment opens at its time with the slider broken into a chapter that starts there. Colours, font and radius come from the --harness-* register. The lightbox opens a cited moment the way a shared link with a timestamp does.

A failed turn never costs the conversation

Session.prefillUserMultimodal writes a warm image, audio or video turn on a fork and fast-forwards the trunk to it, using liblloyal's new fast_forward through lloyal.node 4.1.0. If the prefill fails, for room or a decode error, only the fork is pruned: the trunk, its handle and the user side are exactly as they were. Branch.fastForwardSync is the primitive.

Before any of that, admitted(descriptors, { context }) prices a turn's attachments against the share of the context a harness allows and refuses an over-budget attach with what it would cost and what is allowed, so a reader waits for nothing. useAgent throws SpawnRefused for an agent it cannot seat, extendSpine throws ExtensionRefused, and the spine:refused trace event records both.

Versions

sdk 5.1.0, lloyal-agents 7.1.0, rig 6.1.0, media 0.3.0, ui 0.3.0, binding 0.3.0, desktop 0.4.0, host 0.4.0, relay 0.3.0. Every package requires lloyal.node 4.1.0 or later. The documents ability is 0.3.1 and accepts media 0.3.

Upgrading

• AttachmentStore has three new required members: size, readRange, readWhole. PreparedContent.bitmaps is now inputs.
• MultimodalInput has a video member; code that narrows it exhaustively must handle it.
• admitted takes reserve and free.
• The host accepts a write only from its own pages. A Vite dev proxy with changeOrigin and no xfwd rewrites Host, so a scaffold made before this release needs xfwd: true on its /v1/media, /v1/content and /v2 proxy entries in targets/web/vite.web.config.ts. New scaffolds have it. A production reverse proxy that rewrites Host must set X-Forwarded-Host.
• /v2 admits only video originals by default; pass registry: true to BlobUploads for the full registry.

Ships with lloyal-ai 1.19.0, on liblloyal v3.1.0 and lloyal.node 4.1.0. Documentation: Attachments.

What's Changed

• feat: native video attachments, end to end by @lloyal-research in #149

Full Changelog: v6.0.0-rig...v6.1.0-rig

v6.0.0-rig

Part of release 2026.10.3 (sdk 5.0.0, lloyal-agents 7.0.0, rig 6.0.0, ui 0.2.0; host 0.3.0, desktop 0.3.0, dev-tools 0.6.0)

llama.cpp b11559 downstream, on lloyal.node 4.0.0. The chat parser is an opaque handle carried through the SDK; the context's batch size is the recovery floor; a report owns the room above the floor; the voluntary report cap is opt-in; a stopped report's done event carries its reason and the fold keeps the draft; a salvage trims only a genuinely unfinished escape. The abilities admit the new majors.

What's Changed

• feat: llama.cpp b11559 downstream; reports own the room; a stopped report is shown as stopped by @lloyal-research in #148

Full Changelog: v5.9.0-rig...v6.0.0-rig

v5.9.0-rig

Part of release 2026.10.2 (liblloyal v2.2.0, lloyal.node 3.4.0, lloyal-ai 1.17.0).

rig 5.9.0, sdk 4.2.0, lloyal-agents 6.2.0. A tool declares how its grammar is sampled: grammarOrder: 'sampleFirst' runs the sampler chain on the raw candidates and checks the one chosen token against the grammar, masking the whole vocabulary only on a refusal; mask first stays the default. The SDK exposes Branch.setGrammarOrder, the pool applies the order to the terminal tool on recovery and to a tool once it is identified under the lazy grammar, and citedReport declares sample first. Measured on Fieldnote with three agents, a report step fell from about 500 ms to about 100 ms. The platform depends on lloyal.node 3.4.0.

What's Changed

• fix(agents,sdk): sample first under the terminal tool's grammar by @lloyal-research in #147

Full Changelog: v5.8.0-rig...v5.9.0-rig

v5.8.0-rig

Part of release 2026.10.1 (liblloyal v2.1.0, lloyal.node 3.3.0, lloyal-ai 1.16.0).

rig 5.8.0, sdk 4.1.0, lloyal-agents 6.1.0, binding 0.2.1, desktop 0.2.1, media 0.2.1, ui 0.1.1. The transcription service and its paired provisioning, bounded audio through the content plane, the voice command group, microphone capture with a waveform, the popover, and desktop microphone consent. The platform depends on lloyal.node 3.3.0.

What's Changed

• feat(rig,desktop): a harness can run with no project directory by @lloyal-research in #137
• a boot failure says so instead of a dark screen by @lloyal-research in #138
• every catalogue model has a mirror, not just two of eight by @lloyal-research in #139
• docs: clarify licensing FAQ and sync all FSL package copies by @lloyal-research in #141
• docs: rename App to Ability in the Grant and documentation by @lloyal-research in #142
• docs: correct CLI license references by @lloyal-research in #143
• feat: add resident transcription and voice input primitives by @lloyal-research in #146

Full Changelog: v5.6.0-rig...v5.8.0-rig

v5.6.0-rig — the stable set

The Harness Development Kit's stable set, the service-contract arc's code with the prerelease suffix dropped: sdk 4.0.0 · lloyal-agents 6.0.0 · rig 5.6.0 · binding, host, relay, media 0.2.0 · dev-tools 0.5.0 · ui, desktop 0.1.0, every internal dependency on a caret at its new floor, on lloyal.node 3.2.0. One change since alpha.11: a desktop renderer that subscribes late is told what it missed first, so the dev pane shows on a first launch. Ten packages published with provenance.

What's Changed

• agents: the pool becomes a scheduler by @lloyal-research in #119
• feat(media,rig,agents,abilities)!: documents — PDF ingress, a documents ability, assets on the run by @lloyal-research in #120
• feat(rig): the content route table becomes a part — resolveContent by @lloyal-research in #123
• What an agent received lives on the agent (supersedes #121) by @lloyal-research in #124
• feat(agents,abilities): the tool-lifecycle contract — Tool.hooks, the frame, gates as data by @lloyal-research in #125
• fix(agents,media): release-review corrections — PDF inspection refuses, one owner per fork, heals at the original parent, the two test gates by @lloyal-research in #126
• Composition arc: steps 2–4 — admission and typed output, the recut, and everything below the seam by @lloyal-research in #127
• feat(rig,ui,dev-tools): the dev pane derives its settings from the config table; unhandled commands get their own door by @lloyal-research in #129
• ci: a PR into an arc branch runs CI by @lloyal-research in #130
• feat(agents,rig,ui): prompts and nudges become the app's functions (PromptOf, Budget.nudge); rig/testing, Reports, taskKey, serveDefaults lifted from the template; ui/prose and SpawnDecision.timeline by @lloyal-research in #131
• fix(abilities): wikipedia keeps the contract its siblings already keep by @lloyal-research in #132
• feat(agents,rig): a pool's schema is the grammar of its agents' answers — and a spine's format ends with its body by @lloyal-research in #134
• feat(rig): compose every model beside the trunk as a service by @lloyal-research in #136
• feat!: multimodal integration and a content-addressed media package by @lloyal-research in #118

Full Changelog: v5.5.1-agents...v5.6.0-rig

v3.8.0-rig

What's Changed

• chore(release): reconcile root lockfile with v3.1.1 workspace versions by @lloyal-research in #17
• feat(agents): target-dispatch fan-out — per-agent serial, inter-agent concurrent by @lloyal-research in #19
• feat(agents): fold-based parallel recovery — per-report budget + maxLength cap by @lloyal-research in #22
• feat(agents): graceful wind-down (drain) for the agent pool by @lloyal-research in #23
• fix(agents): export WindDown from the package entry (#23 follow-up) by @lloyal-research in #25
• feat(agents): recovery-as-turn — in-loop parallel recovery via the nu… by @lloyal-research in #26
• chore(release): agents 3.2.0 + rig 3.1.2 (lloyal.node ^3.0.1) by @lloyal-research in #29
• docs(agents,cli): fix stale docstrings + scaffold pin to match current API by @lloyal-research in #30
• chore(release): harness.dev 0.5.1 — ship the scaffold template fix by @lloyal-research in #31
• chore(apps): web + corpus 1.2.0 — first channel release carrying Tool.fanout by @lloyal-research in #32
• fix(rig): lazy parseMarkdown binding load (3.1.3) by @lloyal-research in #34
• chore: lloyal.node devDependency ^3.1.1 by @lloyal-research in #35
• feat(binding): @lloyal-labs/binding — the harness's headless interface by @lloyal-research in #36
• feat(binding): uniform Binding contract + session plane + wss; scaffold relay by @lloyal-research in #40
• chore(release): publish binding + relay; raise lloyal.node peer floor (rig 3.1.4) by @lloyal-research in #41
• fix(relay): add repository field for provenance publish by @lloyal-research in #42
• feat(host): @lloyal-labs/host — the box model-runtime host by @lloyal-research in #43
• ci(release): publish @lloyal-labs/host alongside binding + relay by @lloyal-research in #44
• test(host): real-model B-mechanism integration suite (Metal) by @lloyal-research in #45
• feat(rig): verified project-local model catalog + resolveModel by @lloyal-research in #46
• feat(harness-cli): blank template on the harness contract + verified model fetch by @lloyal-research in #47
• feat(rig): App-role model provisioning via AppManifest.requires by @lloyal-research in #48
• refactor(agents,rig)!: rename AppModelRole → Service; publish agents 3.3.0 by @lloyal-research in #50
• refactor(agents,rig)!: rename AppManifest field requires → services by @lloyal-research in #51
• chore(apps): corpus + web 1.3.0 — carry services + services-aware framework floors by @lloyal-research in #52
• feat(rig): provisionAppModels rerankerLoad opts (tune the shared reranker) by @lloyal-research in #53
• chore(wikipedia): bump 1.0.0 → 1.1.0 for the defineApp(manifest, setup) reshape by @lloyal-research in #55
• feat(harness-cli): blank template — desktop + web targets by @lloyal-research in #54
• feat(rig): ReportTool schema-extension seam (extraProperties/extraRequired) by @lloyal-research in #56
• chore(harness-cli): repoint blank wikipedia dep to 1.1.0 by @lloyal-research in #57
• feat(harness-cli): blank template — canonical served driver + reranker-less Runner contract by @lloyal-research in #58
• feat(harness-cli): research template + create --template flag by @lloyal-research in #59
• feat(harness-cli): interactive create (Ink) + dispatcher hardening + ESM by @lloyal-research in #60
• feat(harness-cli): models: + targets: command suites (0.6.0) by @lloyal-research in #61
• chore(harness-cli): release harness.dev 0.6.0 by @lloyal-research in #62
• chore(harness-cli): 0.6.1 — README rewrite for the 0.6.0 command surface by @lloyal-research in #63
• chore(harness-cli): 0.6.2 — README teaches the one-harness/N-targets model by @lloyal-research in #64
• chore: rename model catalog id reasoning-4b → qwen3.5-4b by @lloyal-research in #65

Full Changelog: v3.1.1...v3.8.0-rig

v3.1.1

What's Changed

• fix(agents,rig): planner thinking-off — fix clarify loop (3.1.1) by @lloyal-research in #15

Full Changelog: v3.1.0...v3.1.1

v3.0.2

Full Changelog: v3.0.1...v3.0.2