v0.8.0

This release brings text generation to Karume: a chat-capable Gemma 4 E2B pipeline — tokenizer, host sampler, streaming output, multi-turn sessions — running entirely on WebGPU, with the conversation window as a run-time knob instead of a table baked into the distribution. It also cuts host RAM during loading by roughly 77–83% (Anima Turbo v1.1 f16: 4,069 → 711 MiB) by capping every shard at 256 MiB, splitting oversized tensors across shards, and reading all shards through a single reused buffer.

This is a breaking release for all three packages. The distribution format changed and every model repository was re-published, so older package versions cannot read the new revisions — see 💥 Breaking Changes below for the migration.

💥 Breaking Changes

• Distribution format: shards are capped at 256 MiB and large tensors are split across them. The cap is now a single constant measured over the whole file (header included — the same number the manifest's size reports), where it used to be 1 GiB measured over the data section with a 1.5 GiB tail allowance. A tensor that does not fit is split along its first dimension into pieces named <tensor>#00001-of-00003, laid out one per consecutive shard, with the quantization scale living in the first piece's shard. The first shard of every component is graph-only (karume_ir, empty data section), so admission still reads a few MB. Old runtimes reject pieces as "extra + missing" tensors, and @karume/hub rejects a manifest that declares a shard over the cap. Migration: re-publish with karume dist (existing series move in place, byte-for-byte, with karume repack), and upgrade @karume/runtime and @karume/hub together.
• All model repositories were re-published in the new layout (hdae/karume-anima, hdae/karume-irodori-v4-small, hdae/karume-sbv2-jvnv, plus the new repositories listed under ✨ Features). The pinned revision constants shipped in this release point at the new layout; packages at 0.7.0 or earlier cannot read those revisions.
• @karume/hub now requires @hdae/fetch-cache 0.6.0 (was 0.5.0). The new version can read a response into a caller-supplied buffer, which is what makes the RAM reduction apply to the HTTP path as well.
• StreamedAsset.bytes / ModelShard.bytes may be a view into a buffer that the next shard overwrites. streamAssets now allocates one buffer per component — sized to that component's largest shard — and hands out a prefix view of it. The bytes are valid only until the next next(). Before: const all = []; for await (const s of streamAssets(...)) all.push(s.bytes); After: all.push(s.bytes.slice()) — or consume each shard inside the loop, which is what createSessionFromShards does.
• @karume/hub: LoadedManifest.hubUrl was removed, and repo / revisionSha are now optional. A manifest can come from a local directory, which has no repository or revision. The same applies to HubFetchError.repo and IntegrityError.repo, and IntegritySource gained a "local" member — a switch over it needs the extra branch. If you were composing URLs from hubUrl, use the file paths from resolveFiles(...) against your own base instead. LoadedManifest is also a class now and can no longer be assembled as an object literal outside @karume/hub.
• @karume/runtime: estimateSessionMemory / estimateGraphMemory require EstimateOptions.maxStorageBufferBindingSize for graphs that use the states attention form. The transient score matrix is now accounted for, and its size depends on how many row blocks the device's binding limit allows, so the estimator refuses to guess. Before: estimateSessionMemory(model, { generation }). After: estimateSessionMemory(model, { generation, maxStorageBufferBindingSize: gpu.limits.maxStorageBufferBindingSize }). Graphs without that form are unaffected.
• @karume/runtime: BatchScope.finish() now throws host-side failures. A non-awaited enqueue that fails before reaching the GPU (host-side validation, recipe construction) used to let the batch settle as a success with one dispatch silently missing; the first such failure is now re-thrown from finish(). Wrap finish() in try / catch (or await it) if you were treating it as infallible.
• @karume/runtime: issuing a second run on a GenerationContext while the first is still outstanding now throws ExecutionError. The KV length stayed correct but the rotary position silently diverged. Await each run before issuing the next one; separate contexts and one-shot run calls are unaffected.
• @karume/runtime: NaN handling in the softmax family changed. A row whose values are all NaN now produces NaN instead of a silent exact 0 (softmax, safe_softmax, and the fused attention statistics use a NaN-preserving row max), and a fully masked row in fused attention now produces exactly 0 instead of an undefined value. Inputs that are neither all-NaN nor fully masked are bit-identical to 0.7.0.
• @karume/runtime: numeric contracts are enforced where they were only documented. eps must be finite as an f32 and at least 2^-126 (a subnormal was flushed to zero on some drivers, resurrecting eps = 0 on the GPU only); attributes declared finite are now checked as f32; conv1d / conv2d with K = 0, out-of-range deform_conv2d geometry, and gru_scan hidden widths outside 1–256 are rejected with OpContractError; shapes whose element count exceeds 2^53 are rejected before allocation. No shipped distribution is affected.
• @karume/models: generateGreedy and GreedySpec were removed from the public surface. The static wiring and the per-request arguments lived in one call, and the GenerationContext lifetime leaked to the caller, which reproduced multi-turn accidents in consuming code. Replacement: Gemma4Pipeline.sequence() → GenerationSequence.generate(...) (or Gemma4ChatSession / chat() for the string-level surface).
• @karume/models: ANIMA_TURBO_CURRENT was removed — use ANIMA_CURRENT. Upstream now publishes Turbo as an official checkpoint, so the LoRA-baked hdae/karume-anima-turbo repository is retired. hdae/karume-anima carries the five official variants with anima-turbo-v1.1 as the default model, and third-party fine-tunes (anima-wai-v1.0, anima-copycat-20260610) moved to the new hdae/karume-anima-extra. Before: fromPretrained(ANIMA_TURBO_CURRENT). After: fromPretrained(ANIMA_CURRENT) — which now resolves to Turbo v1.1 by default; pass model: "anima-v1.0" to keep the previous generation. The i4 variants are no longer shipped for any Anima model, and the Aesthetic variant's default step count is 30.
• @karume/models: Anima produces different pixels at some resolutions. VAE tile placement changed (see 🐛 below), so an edge whose tile count changes is laid out differently: 16 of the 89 edges that 0.7.0 accepted recount (1440, 1504, 1536, 1552, 1600, 1632, 1696, 1728, 1744, 1792, 1824, 1872, 1904, 1968, 2000, 2032 px), which makes about a third of the accepted width × height pairs produce different bytes for the same seed and prompt — 1824×1248 among them (5 → 4 tiles on the long edge). The common sizes are unaffected: 512², 1024², the 1344×768 family and 2048² are bit-identical to 0.7.0. Seams were reviewed by eye across four seeds and two prompts before adopting the new placement, and there is no flag to restore the old layout.
• karume (PyPI) 0.8.0: karume verify / karume dist now reject any shard over 256 MiB (measured as file length). karume repack migrates existing series into the new rule without re-exporting — which matters because re-running GPTQ calibration changes the shipped bytes. The repository's export recipes (outside the wheel) dropped the anima-turbo LoRA-baking pipeline along with the retired repository above.

✨ Features

• Text generation, end to end. New ./gemma subpath in @karume/models: Gemma4Pipeline.fromPretrained(...) / .fromAssets(...), then chat() for string-in / string-out streaming, Gemma4ChatSession for stateful multi-turn conversations that continue the KV cache across turns, or sequence() for the low-level AsyncIterable<GenerationEvent> loop. Sampling runs on the host — temperature, top-k, top-p, repetition penalty, logit bias, seeded RNG — in the same order as transformers' LogitsProcessorList, with temperature 0 degenerating to exact greedy. Stopping is layered: token ids (GenerationRequest.stopTokens, unioned with the distribution's EOS set) at the sequence level and strings (Gemma4ChatOptions.stopStrings) at the chat level, plus AbortSignal everywhere. The tokenizer is a SentencePiece-BPE implementation with a streaming UTF-8 detokenizer, compiled to a distribution asset (32.2 MB upstream JSON → 9.6 MB), with its token ids verified verbatim against the upstream tokenizer over the whole vocabulary.
• New model repository hdae/karume-gemma4-e2b (Gemma 4 E2B, Apache 2.0, ~3.8 GiB): the product graph, the per-layer-embedding sidecar, and the tokenizer asset. The sidecar is loaded lazily under a byte budget (maxResidentPleBytes), so its 2.22 GiB never becomes resident all at once.
• The conversation window is a run-time knob. RoPE tables are no longer baked into the distribution; the host generates the cos/sin rows for each chunk. Gemma4Pipeline.sequence({ capacity }), Gemma4ChatOptions.capacity and Gemma4PipelineOptions.chunkLength are bounded by two limits the distribution declares: chunkLength ≤ maxChunkLength (768 for E2B — the range the prefill graph was traced over, which the distribution has to state because the IR does not carry symbol bounds) and chunkLength ≤ capacity ≤ maxPosition (131,072 for E2B, the model's own declared limit). Both gates run before any bytes are opened, on fromAssets as well as fromPretrained, so a hand-written Gemma4PipelineConfig must declare maxChunkLength. The distribution ships 4,096 / 768 as defaults. Gemma4Pipeline.estimateSessionMemory({ capacity, chunkLength }) reports what a given window will cost before you allocate it (128K estimates at 4,956 MiB peak on an RTX 3080 Ti).
• Local distributions are a first-class fetch source. localDirectory(adapter) in @karume/hub, and denoDirectory(root) in the new @karume/hub/deno subpath, produce an opaque DistributionSource that loadManifest and every family's fromPretrained accept in place of a repository reference — so fromPretrained(denoDirectory("models/karume-gemma4-e2b")) gives the same experience as the hub without an HTTP server, without a CacheStorage copy, and with only --allow-read. Cross-repository component references need an explicit mapping (no guessing). In the browser, pass your own DirectoryAdapter over OPFS or the File System Access API.
• fromAssets can open sharded distributions. The whole-buffer entry points now accept <role>[i] key sequences and stream them through the same shard path as fromPretrained instead of failing with "component missing" — the limitation announced in 0.7.0 is gone. Gaps in the indices or a mix of indexed and plain keys fail loudly. The whole-buffer surface still holds every shard in host RAM at once; the reused-buffer path is fromPretrained.
• Memory admission happens before the download. The exporter derives each distribution's requiredLimits from the actual tensors and state slots; @karume/models reads the adapter's limits (readAdapterLimits — the adapter is read and discarded, since WebGPU lets it expire at any time) and rejects an incompatible device before a single weight byte is fetched. At session build, every weight and state buffer is compared against maxBufferSize / maxStorageBufferBindingSize and all offenders are listed in one error, instead of relying on an out-of-memory error scope that some backends never report.
• New model repositories hdae/karume-irodori-v4.1-small (upstream Irodori TTS v4.1-Small, MIT) and hdae/karume-anima-extra (third-party Anima fine-tunes, borrowing the official repository's text stack by cross-repository reference).
• Civitai intake for Anima fine-tunes. python -m anima.civitai <AIR|URL> resolves a model version through the Civitai API, downloads the primary checkpoint into a versioned directory, verifies its sha256 against the API record before naming the file, and records provenance (AIR, verbatim version name, licence fields, description, timestamp) that the export chain carries forward to the model card. Idempotent on re-run; the download needs CIVITAI_API_TOKEN.
• Sessions on one device share compiled pipelines. The pipeline cache moved to the GPUDevice's lifetime, so rebuilding a session no longer recompiles the same WGSL. Diagnostics split accordingly: pipelineCount is now this session's usage and devicePipelineCount is the device-wide total.
• New diagnostics and hooks: SessionBuildStats (shard wait / decode / buffer create / upload fence, per build), ChunkBudgetStats (submit-chunk budget overshoot), Gemma4PipelineOptions.onRunDiagnostics and onPrefill, and GpuInternalError is now public (the counterpart of GpuOutOfMemoryError for pipeline creation failures).
• BatchScope.settle() closes the timing window mid-batch with a single fence, so a long fence-free batch stops running on the initial chunk-size estimate. Calling finish() while a settle() is still in flight is rejected (its fence would otherwise resolve after the error scope is popped and escape as an unhandled rejection), so await each settle() before closing the batch.
• @karume/models barrel additions: formatResolution / parseResolution / animaLatents / denormalizeLatents were reachable only from ./anima, and parseGemma4PipelineConfig is exported so fromAssets consumers can build a validated config without a cast.

🐛 Fixes

• A load-time AbortSignal no longer kills generation after the load succeeded. fromPretrained's signal was captured by the session-building closure, so an AbortSignal.timeout or an unmount abort made every subsequent generate fail on a fully loaded model.
• karume dist (PyPI) accepts cross-repository references for a multi-model repository. The gate that allowed them only for a single-model build was a leftover from the one-model Turbo repository; hdae/karume-anima-extra carries two fine-tunes that borrow the same text stack, so the build would have failed at release time. The gate now checks what the old one only approximated: every model's local copy of a referenced role must be byte-identical to the referenced repository and resolve to the same reference, otherwise the build is rejected naming the role and both models. Referenced roles are also left out of the shared/ folding, so no unreferenced file remains in the repository.
• A failure inside the chat layer no longer reports a clean stop. chat() and Gemma4ChatSession.send() resolved done from the inner sequence, which is closed with return() when this layer fails — so an error in onPrefill or in the streaming detokenizer (unknown token id, invalid UTF-8) reached anyone reading only done as { reason: "closed" }. That failure is now thrown from done, and the turn is not committed to the KV cache; aborted is still carried through from the inner sequence unchanged.
• A failing dispose() no longer wedges the pipeline. Turns are serialized through a single slot that was released only after the sequence was disposed, so a throwing disposal left the slot taken and every later chat() / send() / dispose() waited forever. The slot is now released unconditionally, with both errors delivered as an AggregateError when the turn failed too, and Gemma4Pipeline.dispose() runs every stage before reporting the collected failures as an AggregateError, so the PLE sidecar's host memory is freed even when an earlier stage throws.
• An overflow policy that drops the turn being answered now fails loudly. A Gemma4ChatOptions.onOverflow returning a history without the newest user turn produced an answer to a question that was no longer in the prompt, with no exception raised anywhere; the returned history is now compared by value (returning copies stays valid) and rejected if that turn is missing. The overflow check also runs before the sequence is created, so a conversation that has to be truncated no longer allocates a full capacity of KV and drops it again.
• Invalid generation requests are rejected at the call instead of at the first token. maxNewTokens, stopTokens and the sampler spec were validated inside the async generator body, so chat() / send() handed back a stream that only threw once pulled — and left the session marked busy in the meantime. The same single validator now runs synchronously when the request is issued.
• Device-loss errors carry the reason the backend reported. GpuDeviceLostError said only where the device was lost and discarded GPUDeviceLostInfo.reason / .message, which is often the only surviving trace of the real cause — errors raised on an already-lost device never reach an error scope. The reason is now appended (… — reason: <reason> / <message>) with the existing prefix unchanged, and the examples/gemma4 script expands SuppressedError / AggregateError / cause chains, of which Deno prints only the outermost shell.
• Graph shard bytes are no longer resident for the pipeline's lifetime. Holding the prepared model kept the graph shards alive — 2,461.6 MiB of external memory for Anima Base f16, the size of the graph shards. Now only the parsed graph is kept (0.7 MiB). The trade-off is that every session build re-reads those shards (+1.8–6.6 s per build on Anima, tracked as perf-ledger L-9).
• Attention no longer takes dp4a on faith. The dot-product instruction was selected from adapter enumeration alone, which says nothing about whether the instruction and its emulation agree on that device (the first version of the check misfired on Apple M2, where the two variants turned out to be bit-identical and the shared f32 epilogue is one ULP from the reference). A device-level canary now runs both variants against a known answer at first use and falls back to emulation for attention (or fails loudly if neither variant is sane); on conforming devices the pipeline keys and the numbers are unchanged, at a one-time cost of ~39–55 ms.
• Silent NaN from tanh under fast-math is fixed. gelu_tanh, tanh and gru_scan called the built-in with unguarded arguments; a backend implementing it as (exp(2x)−1)/(exp(2x)+1) overflows f32 past |x| ≈ 44.36 and returns Inf/Inf = NaN. Gemma 4 E2B's real activations reached that on Apple M2 and produced all-NaN logits on every token. The saturating implementation is bit-identical for every non-NaN value (verified against frozen WAV hashes and the Gemma 4 goldens) and propagates NaN correctly.
• Text assets are validated consistently across families. Duplicate vocabulary lines and merges (previously last-wins), non-integer or out-of-range code points, and non-positive maxLength / samplingRate now fail loudly instead of silently producing a different token sequence or a broken WAV.
• Depth Anything reports non-finite output instead of a black map. A NaN spread through the resize into neighbouring pixels and then vanished into the documented min/max visualization recipe, which rendered a uniform black depth map that looked like a valid result. It now throws with the offending coordinate.
• A shared quantization scale between two weights is detected when the model opens. Two initializers pointing at the same scale passed every gate as long as the channel counts matched, and the later weight was dequantized with the earlier one's scale.
• run / enqueue snapshot their arguments at issue time. The generation handle and copyOutputs were read after the call returned, so rewriting the request object right after a non-awaited issue could split "the value the dispatch count was computed from" from "the value written into the uniform".
• Aborting and failing fast now work for already-cached files. prefetchAssets walked every remaining reference when the cache was warm, because the abort signal was only observed in the network layer. The download byte budget also normalizes URLs the same way the fetch layer does, so a mirror spelled with an explicit default port no longer silently disables the size gate.
• Passing a local path to fromPretrained fails with a clear error instead of being spelled into a Hugging Face URL and coming back as a 401/404 — for spellings that can only be paths (a leading ./, ../ or /, or a trailing /). A bare models/karume-gemma4-e2b still looks like owner/name and is not caught; use denoDirectory(...) / localDirectory(...) for local distributions.
• SBV2 validates seed alongside the other knobs, instead of after building the 334 MB text encoder and running it once.
• Anima accepts eight resolutions it used to reject. VAE tile placement no longer requires the stride to divide the span evenly, so the tile count no longer spikes on spans with few divisors and the guard that rejected those resolutions is gone. Resolutions whose tile count is unchanged produce byte-identical images; the ones that recount do not — see 💥 above.
• The exporter's activation-quantization scale mirrors the runtime. It computed amax / 127 where the kernel computes amax * f32(1/127), leaving a few percent of rows one ULP apart.

⚡ Performance

Measured on an RTX 3080 Ti under Linux/Vulkan unless noted; methods and full tables are in docs/research/ and docs/perf-ledger.md.

• Host RAM while loading a model dropped by roughly 77–83%. Three changes compose: shards are capped at 256 MiB, tensors too large for a shard are split into pieces, and the streaming path reads every shard into one reused buffer instead of allocating per shard. The peak is now roughly "a ~0.45 GB constant plus one shard". Anima Turbo v1.1 f16: 4,069 → 711 MiB. Gemma 4 E2B (i4): 2,622 → 613 MiB. Over HTTP the same holds: Gemma 4 cold 1,740 → 703 MiB, warm 1,408 → 684 MiB; Anima f16 warm 2,242 → 743 MiB. Loading Anima also got faster (11.2 → 5.9 s) because the allocate/collect churn disappeared. On Apple M2 (Metal) the same shard cap on top of the reused buffer took Anima from 3,068 to 1,435 MiB; a 0.7.0-equivalent Mac baseline was not measured, and the remaining gap is believed to come from unified memory counting GPU-side buffers in the process's resident set.
• Gemma 4 decode is 2.6× faster. A GEMV kernel family for the decode step (M = 1, i4 weights) replaces the general GEMM skeleton, which wrote 4 outputs from 64 threads and serialized weight-read latency behind two barriers per K tile: 84.2 → 32.5 ms/token, and the kernel itself 62.4 → 7.4 ms (×8.45). Bit-identical to the previous path.
• Long contexts decode about twice as fast. The attention value-accumulation step now reduces along the KV axis in parallel, which is the default for Gemma4Pipeline: 82.2 → 41.1 ms/token at a 16,000-token prompt, 40.0 → 33.0 ms/token at 992 tokens. The generated token sequence is unchanged; Gemma4PipelineOptions.stateAttentionReduce: "sequential" restores the reference reduction order. The window scales to the model's declared maximum on the same 12 GB card: a 131,000-token prompt prefills in 2,571 s (171 chunks of 768) and then decodes at 112 ms/token.
• Prefill defaults to a chunk length of 768, the maximum the graph was traced over. Against the 32-row chunks used during development the same prompts prefill 5–9× faster: 326 → 60 s for a 16,000-token prompt, 75 → 8.3 s for 4,096 tokens, 19.2 → 2.2 s for 992 tokens, with an identical token sequence. Gemma4PipelineOptions.chunkLength overrides it per pipeline, up to the declared maxChunkLength.
• Per-token activation quantization is 4.1× faster on short rows. Rows shorter than the workgroup (the D=128 stage feeding fused attention, which is 87% of the quantized rows in practice) are now packed several to a workgroup instead of leaving 87.5% of the lanes idle: 123 → 30 ms for those rows, 229 → 135 ms for the stage as a whole, −2% of total GPU time on Anima at 1024² / 4 steps. Bit-identical.
• Anima's VAE decode is 22% faster at 1824×1248 (15 → 12 tiles) as a side effect of the tiling fix above.
• Fence-free batches issue 95% fewer submits. With settle() called once after the first iteration, Irodori's diffusion loop went from ~8,000 submits to 377 for the same work. Wall time is unchanged there — the loop is GPU-bound — but the host-side cost is gone.
• Long prompts no longer scan the whole added vocabulary at every character. Splitting off added vocabulary compared every entry at every character position — 6,415 of them for the EmbeddingGemma tokenizer asset — and now compares only the entries sharing that character as their first, so a position that starts no added token costs nothing.

📝 Docs

• Corrected a security claim in the headers documentation of every family that carried it (seven shipped families; Gemma 4's copy of the same wrong wording was fixed before it ever shipped). It promised that authenticated requests are isolated into their own cache namespace; they are not — headers are passed through to the request, and cached bytes are shared per origin. docs/limitations.md is the normative statement.
• Model cards no longer suggest fromAssets for hub-hosted repositories (fromPretrained is the normal path), and the Anima cards were rewritten around the official / third-party split, including the upstream licence's terms for outputs and derivatives.
• Model cards dropped the per-file table (with the 256 MiB shard cap it had grown to 235 rows on the Anima card); the quant table now carries a Download column instead — the bytes a quant actually fetches, noting the share that is reused across models of the repository and, where it matters, the assets read on the host (Gemma 4's PLE sidecar). Per-file size / sha256 remain in karume.json.
• Model cards of distributions that ship i4 weights now state that the packed int4 storage dtype (I4) is not part of the official safetensors specification and that the official safetensors library (checked with 0.8.0) rejects files containing it; Karume's runtime and exporter read them, files without i4 stay fully compatible, and cards without i4 are unchanged.
• docs/limitations.md gained sections for the run-time capacity / chunkLength knobs and what moving them does to the token sequence, the local fetch source's verification contract (size only — the sha256 is trusted), the by-design ceiling on cross-implementation parity for diffusion pipelines, and the fact that memory preflight compares against absolute device limits only, never against free VRAM.
• On Metal, turning GPU timestamps on (gpuTiming: true, --diagnostics in the examples) takes the device down. On Metal the failure is quantity-dependent, not categorical — a single query set allocates and reports non-zero timings even on macOS 26, but the ~100 sets a Gemma 4 prefill keeps alive at once make createQuerySet fail with Cannot allocate sample buffer; wgpu treats that as fatal, and because errors on a lost device never reach an error scope there is nothing to catch. The earlier attribution — Anima's 3,301 dispatches per step being too large — is withdrawn: the first run is pinned to 16 dispatches per chunk, so even Gemma 4's ~1,500-dispatch prefill keeps roughly 100 query sets alive at once, and GPUQuerySet.destroy() is a no-op in Deno 2.9.x. The repository's own timestamp-query census test therefore fails on macOS as well; it is tracked in docs/known-issues.md. There is no workaround today (reusing a single query set is the candidate fix, with the isolating experiment filed in docs/known-issues.md); examples/gemma4 warns when --diagnostics is passed on macOS, per-op timing needs a Linux/Vulkan machine, and wall-clock timing works everywhere with diagnostics off (the default).

Full Changelog: v0.7.0...v0.8.0