Skip to content

full-rearchitecture-scaffold#16

Merged
gabewillen merged 21 commits into
mainfrom
full-rearchitecture-scaffold
Feb 27, 2026
Merged

full-rearchitecture-scaffold#16
gabewillen merged 21 commits into
mainfrom
full-rearchitecture-scaffold

Conversation

@gabewillen

@gabewillen gabewillen commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

Reopening the full rearchitecture scaffold work on the renamed branch.

  • Branch: full-rearchitecture-scaffold
  • Includes the large rearchitecture scaffold and planner/gbnf updates discussed in-thread.

Copilot AI review requested due to automatic review settings February 26, 2026 00:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Reopens the re-architecture scaffold by pruning generated architecture artifacts, updating build configuration for Boost.SML pinning, and reshaping test/fuzz build wiring.

Changes:

  • Removed many generated architecture markdown + Mermaid diagram artifacts under docs/architecture/**.
  • Centralized the Boost.SML FetchContent repository/tag into cmake/sml_version.cmake and wired it into the top-level CMake.
  • Updated README/AGENTS guidance and adjusted test and fuzz build configuration.

Reviewed changes

Copilot reviewed 93 out of 545 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
docs/architecture/text_tokenizer_preprocessor_wpm.md Removed generated architecture doc artifact.
docs/architecture/text_tokenizer_preprocessor_ugm.md Removed generated architecture doc artifact.
docs/architecture/text_tokenizer_preprocessor_spm.md Removed generated architecture doc artifact.
docs/architecture/text_tokenizer_preprocessor_rwkv.md Removed generated architecture doc artifact.
docs/architecture/text_tokenizer_preprocessor_plamo2.md Removed generated architecture doc artifact.
docs/architecture/text_tokenizer_preprocessor_fallback.md Removed generated architecture doc artifact.
docs/architecture/text_tokenizer_preprocessor_bpe.md Removed generated architecture doc artifact.
docs/architecture/text_jinja_parser.md Removed generated architecture doc artifact.
docs/architecture/telemetry_provider.md Removed generated architecture doc artifact.
docs/architecture/parser.md Removed generated architecture doc artifact.
docs/architecture/logits_validator.md Removed generated architecture doc artifact.
docs/architecture/logits_sampler_token_selector.md Removed generated architecture doc artifact.
docs/architecture/logits_sampler.md Removed generated architecture doc artifact.
docs/architecture/generator.md Removed generated architecture doc artifact.
docs/architecture/gbnf_parser.md Removed generated architecture doc artifact.
docs/architecture/buffer_realloc_analyzer.md Removed generated architecture doc artifact.
docs/architecture/buffer_planner.md Removed generated architecture doc artifact.
docs/architecture/README.md Removed docs index/README for architecture artifacts.
docs/architecture/mermaid/token_batcher.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_tokenizer_preprocessor_wpm.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_tokenizer_preprocessor_ugm.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_tokenizer_preprocessor_spm.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_tokenizer_preprocessor_rwkv.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_tokenizer_preprocessor_plamo2.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_tokenizer_preprocessor_fallback.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_tokenizer_preprocessor_bpe.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_tokenizer.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_jinja_parser.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_jinja_formatter.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_encoders_wpm.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_encoders_ugm.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_encoders_spm.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_encoders_rwkv.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_encoders_plamo2.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_encoders_fallback.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_encoders_bpe.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/text_encoders.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/tensor_lifetime_analyzer.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/tensor_allocator.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/telemetry_provider.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/telemetry_exporter.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/parser_gguf.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/parser.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/model_weight_loader.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/model_loader.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/memory_kv.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/memory_coordinator_recurrent.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/memory_coordinator_kv.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/memory_coordinator_hybrid.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/memory_coordinator.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/logits_validator.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/logits_sampler_token_selector.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/logits_sampler.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/graph_processor.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/generator.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/gbnf_parser.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/decoder.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/buffer_realloc_analyzer.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/buffer_planner.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/buffer_chunk_allocator.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/buffer_allocator.mmd Removed Mermaid diagram artifact.
docs/architecture/mermaid/batch_planner.mmd Removed Mermaid diagram artifact.
cmake/sml_version.cmake Added centralized Boost.SML repo/tag pins.
Testing/Temporary/CTestCostData.txt Removed temporary CTest artifact from repo.
README.md Updated architecture guidance and doc link list.
CMakeLists.txt Wired in sml_version.cmake, updated emel sources, tests list, and fuzz linking on macOS.
AGENTS.md Updated SML rules doc path and expanded state machine conventions.
Comments suppressed due to low confidence (1)

README.md:1

  • The README still links to multiple docs/architecture/*.md files that are removed in this PR (e.g., gbnf_parser.md, generator.md, logits_*, text_jinja_parser.md, and all text_tokenizer_preprocessor_*.md). Update this list to only include docs that still exist after the cleanup, or keep these generated docs in-repo (or generate them in CI) so the links remain valid.
# EMEL

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread README.md Outdated
Comment on lines +14 to +16
> EMEL is currently going through a major re-architecture expected to complete by end of day on
> Monday, February 23, 2026. See [Architecture](docs/architecture/) and
> [Design Docs](docs/designs/) for the latest improvements.
> Monday, February 23, 2026. The source of truth for architecture and design lives in
> `src/emel/**/sm.hpp` docstrings and the generated docs under `docs/architecture/`.

Copilot AI Feb 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warning contains a hard-coded completion date (Feb 23, 2026), which is already stale shortly after it passes and may confuse readers. Consider replacing the date with a status-based statement (e.g., “ongoing”, “in-progress”, or a version/milestone link) so the README doesn’t require frequent edits.

Copilot uses AI. Check for mistakes.
Comment thread cmake/sml_version.cmake
Comment on lines +1 to +2
set(EMEL_BOOST_SML_GIT_REPOSITORY "https://github.com/stateforward/sml.cpp")
set(EMEL_BOOST_SML_GIT_TAG "02cbea023f035185cfb400e6015c981f9b946bae")

Copilot AI Feb 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These pins are set as normal variables, which makes them harder to override for downstream consumers (e.g., when EMEL is used via add_subdirectory, or when testing a different SML revision). Consider defining them as cache variables (e.g., CACHE STRING) so users can override without editing sources, while still providing the pinned defaults.

Suggested change
set(EMEL_BOOST_SML_GIT_REPOSITORY "https://github.com/stateforward/sml.cpp")
set(EMEL_BOOST_SML_GIT_TAG "02cbea023f035185cfb400e6015c981f9b946bae")
if(NOT DEFINED EMEL_BOOST_SML_GIT_REPOSITORY)
set(EMEL_BOOST_SML_GIT_REPOSITORY "https://github.com/stateforward/sml.cpp" CACHE STRING "Git repository URL for Boost.SML")
endif()
if(NOT DEFINED EMEL_BOOST_SML_GIT_TAG)
set(EMEL_BOOST_SML_GIT_TAG "02cbea023f035185cfb400e6015c981f9b946bae" CACHE STRING "Git tag or commit hash for Boost.SML")
endif()

Copilot uses AI. Check for mistakes.
Comment thread CMakeLists.txt
Comment on lines +178 to +179
set(CMAKE_AR "/usr/bin/ar" CACHE FILEPATH "" FORCE)
set(CMAKE_RANLIB "/usr/bin/ranlib" CACHE FILEPATH "" FORCE)

Copilot AI Feb 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forcing CMAKE_AR/CMAKE_RANLIB in the cache mutates global build tool settings for the entire project (and any parent project that includes this one), not just fuzz targets. If the intent is to work around an AppleClang/Homebrew LLVM tooling mismatch, it’s safer to document this as a toolchain requirement or scope it via toolchain configuration variables (e.g., CMAKE_<LANG>_COMPILER_AR / CMAKE_<LANG>_COMPILER_RANLIB) rather than forcibly overriding global cache entries.

Suggested change
set(CMAKE_AR "/usr/bin/ar" CACHE FILEPATH "" FORCE)
set(CMAKE_RANLIB "/usr/bin/ranlib" CACHE FILEPATH "" FORCE)
# Configure language-specific toolchain archiver/ranlib instead of
# overriding global CMAKE_AR/CMAKE_RANLIB cache entries.
set(CMAKE_C_COMPILER_AR "/usr/bin/ar")
set(CMAKE_CXX_COMPILER_AR "/usr/bin/ar")
set(CMAKE_C_COMPILER_RANLIB "/usr/bin/ranlib")
set(CMAKE_CXX_COMPILER_RANLIB "/usr/bin/ranlib")

Copilot uses AI. Check for mistakes.
Comment thread AGENTS.md
Comment on lines 110 to +115
NEVER mutate context in guards.
NEVER read or write context directly from state machine member functions.
ALWAYS keep context focused on machine-owned runtime data required across internal
phase events.
NEVER mirror per-dispatch request/event payload fields into context only for
phase handoff.

Copilot AI Feb 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rule “NEVER read or write context directly from state machine member functions” is ambiguous as written (it may prohibit common patterns like helper/member methods that accept action::context& and are called by actions). Consider clarifying what qualifies as “state machine member functions” here (e.g., methods invoked by SML directly vs. helper functions called by actions) and what patterns are explicitly allowed (such as helper methods that take context/event parameters).

Copilot uses AI. Check for mistakes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 252a8855c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/emel/parser/gguf/actions.hpp Outdated
Comment on lines +24 to +26
ctx.probed = {};
if (ev.requirements_out != nullptr) {
*ev.requirements_out = ctx.probed;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Parse GGUF header before publishing probe requirements

run_probe unconditionally zeroes ctx.probed and copies that to requirements_out without reading the GGUF image, so tensor_count/kv_count/size hints are always reported as 0 even for valid models. In the probe→bind lifecycle this breaks the contract that callers size storage from probe results, which can lead to binding incorrect capacities and later metadata population failures.

Useful? React with 👍 / 👎.

Comment thread src/emel/parser/gguf/actions.hpp Outdated
Comment on lines +45 to +46
void operator()(const event::parse &, context & ctx) const noexcept {
ctx.last_error = EMEL_OK;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Perform real GGUF parse before signaling success

run_parse currently only sets last_error = EMEL_OK and returns, so any non-empty buffer is treated as a successfully parsed GGUF file. This lets malformed or unrelated input advance to parsed with no validated metadata written to bound storage, which can make the loader report success while operating on uninitialized model descriptors.

Useful? React with 👍 / 👎.

@gabewillen gabewillen force-pushed the full-rearchitecture-scaffold branch from c224a8a to ebcf3c5 Compare February 26, 2026 04:18

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

sml::state<initialized> + sml::event<event::load> [guard::cannot_map_parser{}]
/ action::set_invalid_argument = sml::state<errored>,

P1 Badge Preserve load request on invalid parser wiring

The initialized + event::load [cannot_map_parser] branch skips begin_load, so ctx.request is never captured before transitioning to errored. Since publish_error only emits callbacks when ctx.request is non-null, malformed load requests (e.g., missing parser dispatch hooks) fail without invoking dispatch_error or updating request-scoped error outputs, which leaves callers with a silent failure path.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines 28 to 31
ctx.planned_effects = 0;
if (ev.effect_count_out != nullptr) {
*ev.effect_count_out = 0;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Plan real weight-load effects before signaling success

run_plan_load always reports zero planned effects regardless of bound tensors, so the loader pipeline can transition through plan/apply as successful while scheduling no I/O at all. In the normal load flow this means models can be marked loaded without any tensor data ever being requested, producing unusable or uninitialized weights downstream.

Useful? React with 👍 / 👎.

Comment on lines 36 to 37
void operator()(const event::apply_effect_results &, context & ctx) const noexcept {
ctx.last_error = EMEL_OK;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject failed effect application results

run_apply_effects unconditionally sets EMEL_OK and never inspects results/result_count, so failed external operations (for example, effect results carrying non-OK err codes) are silently accepted and the machine advances to ready. This can publish a successful weight-load outcome even when required mappings or transfers actually failed.

Useful? React with 👍 / 👎.

@gabewillen gabewillen force-pushed the full-rearchitecture-scaffold branch from 022eba4 to 836016c Compare February 26, 2026 21:43
@gabewillen gabewillen merged commit e2514cf into main Feb 27, 2026
gabewillen added a commit that referenced this pull request Jul 4, 2026
…dated modeling (codex review round 2) (#90)

* feat: Moshi/PersonaPlex milestone 1 - moshi.cpp reference wiring + moshi GGUF model binding

Registers the "moshi" architecture (Kyutai Moshi / NVIDIA PersonaPlex family)
and wires Codes4Fun/moshi.cpp in as the pinned reference implementation,
following the llama.cpp/whisper.cpp reference patterns.

Model binding (src/emel/model/moshi/):
- component-discriminated contract (moshi.component = lm | mimi | voice)
  with required moshi.* hparams; load_hparams maps into generic params
  (temporal transformer dims, depformer dims, delays, mimi codec shape)
- execution-contract validation per component: family views (lm.*, mimi.*,
  voice.*), hparam-derived tensor shape checks (text_emb/text_linear/emb
  embeddings, RVQ codebooks), per-layer block presence for both fused
  (in_proj_weight) and pre-split (in_projs.N) attention layouts
- raw metadata-less moshi.cpp GGUF caches are rejected explicitly (no
  general.architecture -> loader error); conversion is required

Converter + fixtures (tools/bench/):
- moshi_gguf_convert.py: raw moshi.cpp GGUF -> emel-enriched GGUF; restores
  CRC-mangled tensor names (>=64 chars, moshi.cpp loader.h tensor_name zlib
  crc32 low-nibble scheme), attaches moshi.* hparams from
  personaplex-config.json / pinned presets with tensor-shape cross-checks,
  embeds the SentencePiece text tokenizer as tokenizer.ggml.* (stdlib-only
  ModelProto wire parser), streams tensor payloads through byte-identically.
  Verified against the real HF artifacts: mimi-e351c8d8-125.gguf (all 64
  mangled names restored, data region sha-identical) and voices/NATF0.gguf.
- moshi_make_tiny_fixture.py: deterministic tiny fixtures generated through
  the maintained converter path; committed under tests/models (LFS) with
  provenance in tests/models/README.md

Reference wiring:
- tools/bench/moshi_reference_ref.txt pins moshi.cpp @ f1fabbd14a50
- EMEL_BENCH_NEEDS_MOSHI FetchContent gate in tools/bench/CMakeLists.txt
  (source-only populate; reference driver targets land with milestone 2)
- scripts/setup_moshi_cpp_reference.sh: clone+pin moshi.cpp, sha256-pinned
  downloads (PersonaPlex-7B q4_k, mimi codec, tokenizer, voice prompt,
  config), deterministic 24 kHz WAV fixture, auto-runs the converter for
  the EMEL-lane enriched GGUFs
- scripts/check_domain_boundaries.sh: forbidden moshi/mimi runtime roots +
  generic-recognizer leak checks

Tests: tests/model/moshi/binding_tests.cpp (registry, enriched lm/mimi/voice
fixture loading incl. embedded-vocab SPM tokenizer, raw-cache rejection,
hparam contract negatives, validation negatives); fixture manifest checks.
Full suite 1132/1132 passing; coverage lane passed; lint snapshot updated
with consent (fixture_manifest_tests.cpp now format-clean).

Known pre-existing gate failures (unrelated, documented): paritychecker
generation baselines for LFM2.5-230M-Q8_0 missing on this lineage;
bench_snapshot missing aarch64 entries on x86 host plus uncommitted
decode_wavefront/flash-attention baselines from earlier branch work.

* feat: kernel compute for get_rows/rope/im2col/conv_transpose_1d/norms + silu/elu/gelu/tanh (Moshi M2 step 1)

Lands real scalar compute for the kernel ops the Mimi codec and Moshi LM
need; all were previously enumerated in EMEL_KERNEL_OP_EVENT_LIST and wired
into both backend machines but silently uncomputed.

Every dtype/mode variant is modeled as an explicit guard-selected transition
row (op_unary pattern) - no runtime variant branches inside actions/detail:
- op_get_rows: six variant rows per backend (f32/f16/bf16/q4_0/q8_0/q4_k
  sources; row dequant via the existing quant helpers, index-validated in
  the guard path, unsupported dtypes reject explicitly)
- op_rope: norm/neox pairing rows (ggml op_params layout, ext_factor==0,
  positions i32; freq_base/freq_scale/attn_factor honored)
- op_im2col: f32/f16 destination rows (1D, ggml operand contract; conv1d =
  im2col + existing mul_mat)
- op_conv_transpose_1d: f32/f16 weight rows (ggml contract: p0==0, d0==1)
- op_rms_norm / op_norm: generic scalar path (single algorithm; eps from
  op_params, double accumulation like the reference)
- op_unary: new tanh/elu/gelu/silu subop rows in both backends (gelu uses
  the ggml tanh-approximation constants)

detail additions: dtype_i32/dtype_bf16 codes + sizes, bf16->f32 conversion,
ggml-layout op_param readers, valid_variant_guard combiner, per-variant
compile-time exec functors shared by both backends.

Tests: per-op positive/negative cases in tests/kernel/x86_64_tests.cpp
(gather f32+q8_0 with out-of-range index rejection, rms/layer norm math,
rope norm+neox rotations, im2col columns, conv_transpose_1d accumulation);
unary subop coverage in both backend test files now asserts the new subops
compute and uses sigmoid as the still-unsupported reject probe.
Full suite: 1136/1136 passing. Lint dirty-list shrinkage (6 files now
format-clean) applied with explicit consent.

ggml parity cases in tools/paritychecker land next with the mimi codec
parity lane.

* feat: ggml kernel parity cases for the Moshi M2 op group

Adds paritychecker kernel-engine cases comparing the emel backend machines
against ggml for: unary tanh/elu/silu (tight tolerance) and gelu (dedicated
fp16-table slack), rms_norm, norm, get_rows over f32 tables and over
q8_0/q4_0/q4_k blocks quantized by ggml itself (identical block bytes in
both lanes), rope norm+neox (ggml_rope_ext params), im2col (1D), and
conv_transpose_1d (shared f16 kernel taps).

Two defects the new cases caught, now fixed:
- has_required_src0 rejected sub-byte quantized get_rows sources because the
  generic layout check truncates q4 per-element sizes to zero; op_get_rows
  now shares the native-quantized row-layout contract with mul_mat.
- conv_transpose_1d's f16 variant now rounds input samples through fp16
  before the tap multiplies, matching the reference f16 operand pipeline
  exactly.

scripts/paritychecker.sh --runner=kernel: kernel parity ok (both backends).
Full unit suite: 1136/1136 passing.

* feat: Mimi codec compute layer + broadcast-row kernel variant (Moshi M2 step 2)

Lands the complete Mimi codec numeric pipeline as kernel-dispatched compute
under src/emel/speech/codec/mimi/ (detail layer; the codec actors and facade
land next):

- plan/size/bind: fixed mimi_v0_1 SEANet topology tables cross-checked
  against tensor shapes; one-time weight canonicalization (f16->f32,
  prepare-time transposes so no runtime GEMM transposes); RVQ nearest-search
  tables with the -|c|^2/2 bias column folded in; streaming-state arena
  carved with per-layer offsets; caller-owned arenas sized by required_*
  functions (allocation-free dispatch).
- compute helpers (all numeric work as kernel op events, generator pattern):
  streaming causal conv (left-context state, im2col + mul_mat + broadcast
  bias), streaming transposed conv with overlap-add tail state (full and
  explicit depthwise variants - the caller picks the variant, no shape
  sniffing), SEANet stacks with resnet residuals and ELU, per-token codec
  transformer (fused-QKV, rope, ring-buffer KV with two-segment windowed
  attention, softmax scale param, layer scales, GELU MLP), split-RVQ encode
  (mul_mat_argmax + get_rows + sub residual) and decode (get_rows sums +
  output projections).
- new kernel variant: op_add/op_mul BROADCAST-ROW (src1 = one dst row),
  modeled as explicit guard-selected transition rows in both backend
  machines; elementwise SIMD guards now claim only equally-sized operands
  (scoped to add/mul so mul_mat routing is untouched). Unit tested including
  the mismatched-row rejection.
- tests/speech/codec/mimi_detail_tests.cpp: full encode chain (1920-sample
  frame -> seanet -> 25 Hz transformer -> downsample -> 4 RVQ codes) with
  streaming-state advance and exact reset determinism, full decode chain
  (codes -> depthwise upsample -> transformer -> seanet -> 1920 finite
  samples), out-of-range code rejection.
- tests/models/mimi-tiny.gguf regenerated with the full tiny-scale topology
  (README hash + provenance updated; deterministic generator).

Full suite: 1139/1139 passing. Lint clean; domain boundaries clean.

* feat: Mimi codec actors and facade (Moshi M2 step 3)

Adds the stateforward.SML actor layer over the verified Mimi compute
pipeline, completing the codec component per the approved plan:

- emel::speech::codec::mimi::encoder::sm - one 80 ms PCM frame -> one
  12.5 Hz latent column through explicit frontend/transformer/downsample
  stage states; validation (runtime bound, request shape, buffer capacity)
  and stage outcomes are guard-routed decision states over the per-dispatch
  runtime ctx, whisper-encoder pattern.
- emel::speech::codec::mimi::quantizer::sm - split-RVQ latent <-> codes with
  separate explicit encode/decode state chains.
- emel::speech::codec::mimi::decoder::sm - latent -> PCM through depthwise
  upsample / transformer / SEANet backend stage states (the depthwise
  transposed-conv variant is selected by the transition, not sniffed).
- emel::speech::codec::mimi::sm facade - owns the bound runtime, streaming
  state, and the three child machines (child data owned by parent context,
  cross-machine communication only via process_event); sequences
  encode_frame (frontend -> quantize) and decode_frame (dequantize ->
  backend), handles initialize/reset_stream, and answers pre-initialization
  requests with explicit not_initialized errors. Caller owns all arenas
  (initialize event spans, sized by the detail required_* functions), so the
  actors allocate nothing during dispatch.
- emel::MimiCodec alias in machines.hpp; public facade header any.hpp.

tests/speech/codec/mimi_lifecycle_tests.cpp drives the public facade against
tests/models/mimi-tiny.gguf: initialize -> encode_frame -> decode_frame with
state inspection (session_ready via is()/visit_current_states), reset_stream
code-replay determinism, not-initialized rejection, and undersized-arena
bind failure landing back in uninitialized.

Full suite: 1142/1142 passing. Lint and domain boundaries clean.

* feat: mimi EMEL parity runner verified on the real Mimi model (Moshi M2 step 4a)

- tools/bench/speech/mimi_emel_parity_runner.cpp: EMEL-lane runner (links
  only emel; the reference lane stays a separate executable per the
  reference policy). Loads an enriched mimi GGUF, streams a 24 kHz mono s16
  WAV through emel::speech::codec::mimi::sm one 80 ms frame at a time,
  emits `frame=<n> codes=...` lines, and optionally decodes the code stream
  back to raw f32 PCM (--decode-out). Registered in tools/bench CMake.
- codec bind fix for real GGUFs: GGUF collapses trailing size-1 dims (the
  decoder's final 64->1 conv stores as [3, 64]), so the plan walk now
  threads the running channel count through both chains and derives out
  channels from element counts; every layer's geometry is plan-resolved and
  cross-checked (encoder must land on mimi.dim, decoder on 1 channel).
- paritychecker: op_add/op_mul broadcast-row cases vs ggml repeat-broadcast
  semantics (kernel parity lane green on both backends).

Verified against the real converted mimi-e351c8d8-125 model: a 1 s 440 Hz
tone encodes to 12 frames x 16 code streams with the semantic stream stable
in steady state, and decoding the codes reconstructs a 440 Hz tone at
matching RMS with ~17 dB steady-state SNR - functionally correct end to end
on real weights (scalar throughput ~250 ms/frame; performance work is the
bench suite's follow-up).

Full suite: 1142/1142 passing. Lint clean.

* feat: two-lane Mimi parity infrastructure + speech_codec_mimi bench suite (Moshi M2 step 4b)

Reference lane:
- tools/bench/speech/moshi_reference_driver.cpp: pinned moshi.cpp + ggml +
  SentencePiece driver with deterministic frame-level encode/decode
  subcommands emitting the same `frame=<n> codes=...` format as the EMEL
  runner. Builds against the llama.cpp-pinned ggml (no API skew); C++20;
  SentencePiece v0.2.0 via FetchContent, reference lane only.
- tools/bench CMake: speech_codec_mimi added to the NEEDS_LLAMA list (ggml
  only) and the driver target gated on EMEL_BENCH_NEEDS_MOSHI.

Comparison surface (mimi_compare/v1):
- tools/bench/mimi_compare.py runs both lanes as separate subprocesses
  (two-lane isolation) and reports per-frame code match, token_exact, and
  decode-vs-decode PSNR; reference_backends/moshi_cpp_mimi.json manifest;
  scripts/bench_mimi_compare.sh wrapper (--build-only/--run-only), and
  setup_moshi_cpp_reference.sh --build-only now delegates to it.

Bench suite:
- speech_codec_mimi (tools/bench/speech/codec_mimi_bench.cpp, registry
  33->34): facade frame latency on the tiny fixture - encode 2.53 ms,
  decode 2.97 ms, roundtrip 5.48 ms per 80 ms frame budget. Baseline
  snapshots deferred pending consent.
- dependency manifest records for the suite (txt + cpp table).

Honest parity status, embedded in the compare report and backend manifest:
the reference executes ggml's f16 conv pipeline (moshi.cpp requires f16
conv weights and aborts on an f32 model; attention KV cast to bf16) while
the EMEL lane computes in f32 after lossless f16->f32 canonicalization.
Different effective operand classes, so results are reported as similarity
(25.5% code match, 21.3 dB decode PSNR on the 1 s tone), NOT kernel parity.
Token-exact parity requires the EMEL codec to consume the f16 operand
class end to end - tracked as the next M2 work item.

* fix: scope forbidden moshi speech root to the directory (speech/moshi/)

The reference-lane driver tools/bench/speech/moshi_reference_driver.cpp is
legitimate (reference naming is confined to tools/bench per policy); the
boundary check must forbid the speech/moshi/ runtime DIRECTORY root without
matching that filename. speech::moshi namespace remains forbidden.

* chore: record speech_codec_mimi bench baselines (user-approved snapshot)

Tiny-fixture facade frame latency on this host: encode 2.53 ms, decode
2.97 ms, roundtrip 5.48 ms per 80 ms frame.

* feat: ggml-exact f16xf16 mul_mat kernel variant (Moshi M2 f16 parity step 1)

First piece of the user-approved f16 operand-class pipeline for token-exact
Mimi encode parity:

- kernel detail: can_run_mul_mat_f16 / run_mul_mat_f16 with ggml's layout
  (src0 f16 [k,m], src1 f16 [k,n] both k-fastest, dst f32 [m,n]) and
  vec_dot_f16_ggml - an exact port of the pinned ggml_vec_dot_f16 x86
  AVX2+F16C+FMA path: four 8-lane f32 accumulators over 32-element steps,
  pairwise reduce (0+=2, 1+=3, 0+=1, 128-lane add, two hadds), and a
  double-precision scalar tail; scalar fallback matches ggml's no-SIMD path.
- both backend machines: explicit guard-selected transition row
  (valid_op_mul_mat_f16 -> exec_scalar_op_mul_mat_f16) ahead of the generic
  mul_mat row; generic f32/quantized guards unchanged.
- paritychecker: f16xf16 mul_mat case vs ggml_mul_mat requiring BIT-EXACT
  equality (k=70 exercises the 32-wide loop plus tail) - green on both
  backends.

Full suite: 1142/1142. Next: rebind mimi conv/proj weights as raw f16 with
f16 im2col dispatch, then bf16 KV rounding, then rerun mimi_compare
expecting code match -> 100%.

* wip: streaming weights milestone - phases 1-3 in progress

- SML pin -> external-completion branch (stateforward/sml.cpp PR #16)
- emel aliases + sm_external_completion_tests (9 cases green)
- io/mmap advise_mapping surface (sequential/willneed/dontneed, guard-routed
  kinds, 6 cases green)
- model/tensor/window machine WIP (detail/errors/events/context/guards/actions)
- bench regressions warn by default (EMEL_BENCH_STRICT_REGRESSION=1 for strict)
- LFM2.5-230M fixture graduated to current_publication + 4 parity baselines
- SYSTEM include for sml dep + pedantic pragma guards (g++ coverage lane)

* feat: f16 operand-class codec pipeline + bf16 KV rounding (Moshi M2 f16 parity step 2)

Rewires the Mimi codec onto the reference f16 operand class for models that
store f16 conv weights (the real Mimi model):

- plan/bind: operand class detected on the first encoder conv and recorded
  as the bound property codec_runtime::conv_f16; raw f16 taps and quantizer
  conv1x1 projections are kept alongside the f32 canonical copies
  (prepare_raw_f16, arena sized for both); every conv in an f16 model must
  itself be f16 or bind fails.
- compute: compute_seanet_stack / compute_streaming_conv / compute_rvq_
  encode / compute_rvq_decode are now template<bool conv_f16>. The f16 path
  reproduces ggml_conv_1d exactly: f16 im2col (input rounded RNE) followed
  by the bit-exact f16 mul_mat over the raw f16 weight rows (the ggml
  reshape layout equals the GGUF layout, so weights are consumed verbatim);
  quantizer projections round their input through f16 and use the same
  matmul (ggml conv1x1 semantics).
- actors: operand-class selection is modeled explicitly - new variant
  decision states + guard_conv_f32/guard_conv_f16 rows in the encoder
  (frontend, downsample), decoder (backend), and quantizer (encode, decode)
  machines; effects are templates instantiated per row.
- codec transformer: K/V ring values round through bf16 (ggml
  fp32_to_bf16 RNE port, new in kernel detail) matching the reference's
  CACHE_BF16 cache.

Real-model evidence (440 Hz tone vs moshi.cpp reference): frame 0 code
match 15/16 (was 11/16 on the f32 pipeline); the remaining divergence is
attention numerics (reference converts q and the softmaxed weights to bf16
and dots in physical slot order) plus ggml's fp16-LUT GELU - both mapped
and queued as the next steps toward token-exact.

Full suite: 1142/1142. Lint clean.

* model/tensor/window: streaming weight window machine

Tensor-owned residency for models larger than RAM: whole-file mmap source,
budget-guarded passthrough vs streaming decision, ring of pre-allocated layer
slots filled by a 2-lane I/O pool via staged_read (one actor per slot for
single-writer under concurrent loads), and coroutine suspension on slot loads
through the external-completion co_sm policy.

Acquire is a three-dispatch protocol (resolve/settle/publish) driven
unconditionally by the typed wrapper: resolve joins a slot still mid-load for
a different layer (non-sequential access), settle decides
resident/loading/unscheduled and submits+requires the target, publish routes
on the committed residency, publishes the slot view, and advances the
prefetch ring. The prefetch guard never targets the just-published slot (the
pass wrap can alias them when layer_count % slot_count != 0) and willneed/
dontneed advise flows ahead of and behind the window.

Tests: lifecycle (bind validation, passthrough, rebind rejection, not_bound/
not_streaming legs, allocation-free acquire) and streaming (content integrity
across ring eviction, cold jumps joining busy slots, unbind draining
in-flight loads).

* feat: TOKEN-EXACT Mimi encode parity vs moshi.cpp (Moshi M2 f16 parity step 3)

Completes the exact reference-numerics port. Encode is now token-exact
against the pinned moshi.cpp reference on the real Mimi model: 192/192
codes on a 1 s tone and 592/592 codes on a 3 s chirp+noise+silence torture
signal (37 frames), with lane-vs-lane decode PSNR of 78 dB. The official
compare gate (scripts/bench_mimi_compare.sh -> mimi_compare.py) now runs
with --require-token-exact, and the backend manifest's operand-class
caveats are replaced with the token-exact parity statement.

Exact ports added to the kernel detail layer (all validated in the parity
checker or end-to-end):
- vec_dot_f32_ggml / vec_dot_bf16_ggml: ggml's AVX2 accumulator layouts
  and reduce orders (f32: 4x fmadd + pairwise/hadd reduce, float tail;
  bf16: 4x mul+add with <<16 lane loads, movehl/movehdup reduce, double
  tail).
- v_expf_ggml + soft_max_row_ggml: ggml's vectorized expf polynomial and
  row softmax (per-8 horizontal reduce into a double sum, libm expf tail,
  1/sum in double).
- norm_row_ggml: ggml_compute_forward_norm's row math (float mean from a
  double sum, AVX2 centered-variance blocks, double variance sum).
- GELU now replicates GGML_GELU_FP16 exactly (fp16-quantized input/output
  around the tanh approximation with +-10 saturation); the gelu parity
  case is tightened from 2e-3 slack to bit-exact and passes.

Codec transformer rewritten on those primitives:
- reference rope: timestep-embedding angles (expf(-logf(mp)*j/half)) with
  adjacent-pair rotation and moshi's half-split output layout (the layout
  changes vec_dot accumulation order, so it is part of the contract).
- reference attention: K/V rings hold raw bf16 (ggml CACHE_BF16); q and
  the softmaxed weights round to bf16; scores run over the full ring in
  physical slot order with -inf masking; the V pass materializes the
  transposed bf16 value cache exactly like ggml's cont(transpose(v)).
- transformer projections consume raw row-major weights via the ggml f32
  dot (prepare_matrix_raw replaces the transposed packing).

Throughput snapshot (37-frame real-model encode): EMEL 3.20 s wall /
2.61 s user single-threaded (~86 ms/frame) vs reference 1.23 s wall /
3.24 s user multi-threaded - EMEL already spends ~20% fewer total CPU
cycles for identical output; parallelization is the next lever. Tiny
fixture bench improved slightly (encode 2.40 ms vs 2.53 ms baseline).

Full suite: 1154/1154. Kernel parity green. Lint and boundaries clean.

* gate fixes: benchmark marker for window sm, pedantic pragmas for all sm wrappers

* perf: direct data-plane staging copies in the mimi codec (throughput step 1)

Profiling the real-model encode showed the top cost was not compute but
the staging copies routed through generic strided kernel views (per-element
tensor_offset_bytes/is_dense_contiguous calls dominated the profile). The
conv/convtr staging, ring/overlap state saves, and resnet residual copy are
pure moves with no arithmetic, so they now use direct loops and memcpy in
the codec detail layer.

37-frame real-model encode: 3.17 s -> 2.69 s wall (-15%), 2.59 s -> 2.11 s
user, single-threaded ~73 ms/frame (80 ms real-time budget). Token-exact
parity vs the moshi.cpp reference re-verified after the change (592/592
codes on the torture signal). For contrast the reference burns 3.24 s user
for the same output; the next throughput lever is fork/join lane
parallelism reusing the generator's view-sliced pattern.

Full suite: 1154/1154. Lint clean.

* text/generator: streamed decode route via tensor window

One engagement guard (guard_decode_stream_window_ready, mirroring the
lane-pool capability gate) composed onto the scalar kernel and
native_quantized flash decode routes; streamed rows sit above their
resident/parallel siblings and dispatch window_mode::streamed kernel
instantiations. run_layer's streamed prologue acquires the layer's slot view
from the tensor window (suspending on in-flight loads via the
external-completion drain) and rebases the block's matmul weight records
onto the slot in canonical role order; prefill stays resident through the
always-valid mmap source views (v1 streams decode only).

Owner-side wiring: generator ctor overload takes the bound window actor and
its bind-reported streaming_active; the initializer re-attaches both after
prepare() reconstructs the backend and records the pristine per-layer weight
records the rebase clones from (branch-free bookkeeping).

Tests: streamed-vs-resident argmax token parity through the public generator
path (2 slots over 4 layers), slot-byte consumption proof (model B's file
streamed under model A's records), passthrough disengagement.

* feat: q8_0-quantized Mimi variant - converter + explicit quantized operand class

moshi.cpp cannot run a quantized Mimi at all (its conv path requires f16
weights), so this is an emel-owned operand class.

Converter (tools/bench/moshi_gguf_convert.py --quantize q8_0, needs numpy):
- quantizes the transformer projections (in/out_proj, linear1/2) and RVQ
  input/output projections to q8_0 row blocks matching the emel/ggml
  encoder semantics (per-32 block, fp16 scale, round-half-away); conv1x1
  projections stored [1, in, out] reshape to [in, out] so the quantized
  row length is the real contraction length. Norms, biases, layer scales,
  codebooks, convs, and convtrs keep their float dtypes.
- 347 MB -> 199 MB (-43%), 68 tensors quantized, quantization recorded in
  moshi.mimi.quantization metadata.

Codec: two new independent bound operand properties, each selected by
explicit guard rows per the behavior-modeling directive (never helper
branches): proj_q8 (transformer projections; encoder/decoder gain a
transformer variant decision state with guard_proj_f32/guard_proj_q8 rows)
and rvq_q8 (quantizer projections; the quantizer's variant decision states
now carry three mutually exclusive class rows: f32 / reference-f16 / q8).
Pre-quantized weights bind once into the arena (prepare_raw_q8_0) and the
hot path dispatches the existing AVX2/NEON quantized mat-vec kernels - no
packing or weight conversion in the frame loop.

Real-model quality vs the f16 token-exact reference (37-frame torture
signal): 87% code match, semantic stream 35/37, decode PSNR ~31 dB; an
RVQ-projections-in-fp variant measured identical code match, confirming
the flips come from transformer-latent perturbation inherent to
quantization. The f16 lane keeps its token-exact gate; the q8 artifact is
a size/bandwidth option with these measured bars.

Full suite: 1154/1154. Lint clean.

* refactor: resolve codex review findings (host kernel kind, compile-time
topology unroll, event-payload observation)

Addresses tmp/codex.review.md:

- HIGH kernel_kind: codec_runtime::kernel_kind now defaults to
  emel::kernel::detect_host_kind() (aarch64 hosts route to the aarch64
  kernel actor) and the bound kind is applied to the nested kernel machine
  exactly once at bind; the five per-helper set_kind calls are gone, so no
  backend selection happens inside compute helpers.

- HIGH seanet block-kind branching: compute_seanet_stack's runtime
  layer.kind dispatch is replaced by a compile-time unroll of the constexpr
  topology tables - compute_stack_step<conv_f16, kind> is instantiated per
  table slot (kind is a template parameter resolved from
  k_encoder_topology/k_decoder_topology at compile time), and the public
  entries are compute_seanet_encoder/compute_seanet_decoder chosen by the
  already-distinct actor effects. Only compile-time conditionals remain in
  the stack. Token-exact parity re-verified after the change (37/37 frames
  vs the moshi.cpp reference).

- MEDIUM context-reading wrapper accessors: removed frame_samples()/n_q()/
  frames_encoded()/frames_decoded()/frames_quantized()/frames_dequantized().
  Callers observe the modeled events instead: initialize_done already
  carries frame_samples and n_q, and the done callbacks count completed
  frames (runner, bench suite, and lifecycle tests updated). The child
  actors keep no persistent state, so their contexts are now empty structs
  per the context rules.

- HIGH stage_ok routing: kept, with the pattern now documented at the ctx
  definition - it is the repository's sanctioned per-dispatch outcome
  channel (identical to gguf loader actions writing ctx.err from
  detail::probe_requirements and guards routing on it, and to the whisper
  encoder's runtime ctx). Actions record outcomes of already-executed work
  without branching; guards - pure predicates of the runtime event - route
  the explicit success/error transitions; the mark_*_failed effects assign
  typed errors on the explicit error transitions.

Full suite: 1154/1154. Lint, boundaries clean. Token-exact gate green.

* bench: weight_streaming 3-lane suite + cgroup memory-cap wrapper

New opt-in suite (EMEL_BENCH_WEIGHT_STREAMING=1, set by
scripts/bench.sh --suite=weight_streaming) comparing three lanes on the
LFM2.5-230M-Q8_0 fixture at max_tokens 32 with separate first_token rows:
emel window (streamed decode via the tensor window), emel mmap (passthrough
whole-file mapping, OS demand paging), and a llama.cpp mmap reference at
matched threads. EMEL lanes are EMEL-owned end to end; the reference lane
never shares objects. Note fields record bytes_read and major_faults.

scripts/bench.sh gains --memory-max=<bytes|NNpct|none> (weight_streaming
only): systemd-run user-scope wrapper with MemoryMax + MemorySwapMax=0,
probe with hard fail when delegation is missing, NNpct derived from the
fixture size, run-only mode reusing the default bench build dir.

Known limits recorded in the bench source: EMEL session arenas size to the
model's advertised context_length (3.57GB RSS on this fixture), so honest
capped runs need a larger fixture or a session context cap; reference rows
pair by case name so the llama measurement appears under both lane names.

* scripts: RAM-aware build parallelism clamp

Every build script used bare 'cmake --build --parallel' (one job per core).
The SML template TUs cost 2-4GB of compiler RSS each, so 30 jobs on a 58GB
host oversubscribes memory ~2x and swaps the machine - worst under the
coverage lane's g++ -O0 --coverage rebuild inside quality gates.

scripts/build_jobs.sh computes jobs = min(cores, MemAvailable / 4GB)
(override: EMEL_BUILD_JOBS) and exports CMAKE_BUILD_PARALLEL_LEVEL; sourced
by build_with_zig, bench, test_with_coverage, and quality_gates so the
gate's inner builds are bounded too. AGENTS.md records the rule plus the
no-concurrent-jobs invariant for shared hosts.

* scripts: per-job memory budget for the build clamp

Measured compiler RSS is ~3-4GB per template TU under zig -O3 but ~7-8GB
under g++ -O0 --coverage, so the flat 4GB budget still let the coverage lane
reach the memory ceiling at 8 jobs. The clamp now takes a per-job budget
(EMEL_BUILD_JOB_MEM_GB, default 6) and the coverage build recomputes its own
bound at 8GB/job (override: EMEL_COVERAGE_BUILD_JOBS).

* coroutine-plan: record Phase 5 external-completion completion via streaming weights

* refactor: resolve codex review round 2 - guard-validated non-failing actions, wavefront outcome-slot guard, total submit_or_run, public mimi sizing

Finding 1 (stage_ok routing): every success/error route in the mimi facade
and encoder/decoder/quantizer children is now decided by pure validation
guards BEFORE the corresponding action runs; compute actions are non-failing
by contract and chain unconditionally. detail gains validate_codec_contract
(compile-time dry-run instantiation of the same bind walk, so the validator
cannot drift from the bind it authorizes), validate_codes_in_range (the only
data-dependent per-frame failure), void compute entries, and O(1) bound
sizing (runtime.workspace_floats/frame_floats replace a per-frame model
re-plan in capacity guards). Bind splits into contract -> capacity -> bind
states with distinct bind_failed / arena_capacity errors; frame requests
route explicit request_shape / code_range errors.

Finding 2 (shared outcome slot race): parallel wavefront dispatch now
requires pairwise-distinct caller-owned accepted slots
(all_lane_outcomes_distinct); aliased slots route through the serial path.
Reproduced first with a failing test (misreported failed_lane).

Finding 3 (fallback selection in action): new total fork primitive
thread_pool_scheduler_ref::submit_or_run executes each task exactly once
inside the join window (worker or calling thread - scheduler-internal
capacity handling, not a behavior choice); decode_wavefront action and the
generator sliced-matmul helper are straight-line now.

Finding 4 (detail reach-in): public arena-sizing contract in mimi any.hpp
(prepared/state/workspace/frame_arena_floats); bench + parity runner
migrated off mimi::detail.

Coverage debt closed: new committed operand-class fixtures
mimi-tiny-f16.gguf (f16 convs + RVQ projections) and mimi-tiny-q8.gguf
(dim-32, converter-quantized q8_0 projections) drive the guard_conv_f16 /
guard_proj_q8 / guard_class_q8 rows and their compute paths in tests
(mimi component coverage 66% -> ~90%, detail.cpp 92%).

Also: merge fallout fixes (bench registry 34->35, speech_codec_mimi
reference-lane placeholder records via the sortformer recorded pattern,
per-lane case dump on count mismatch), benchmark markers on the new mimi
sm.hpp files, gcc -Werror class-memaccess sweep in tests (memset on the
non-trivial model::data::vocab replaced with value-init; the encoder test
vocab_builder now reproduces its all-zero contract explicitly).

Verification: full suite 1185/1185; token-exact mimi encode parity vs
moshi.cpp re-verified post-refactor (37/37 frames, decode PSNR 80.7 dB);
bench_snapshot (speech_codec_mimi) and paritychecker lanes green.

* test: drive mimi child-actor validation error routes directly

The facade pre-validates child requests, so the children's explicit
runtime_unbound / request_shape / buffer_capacity / code_range transitions
only execute when the actors are dispatched directly. Covers the remaining
changed-line clusters flagged by the coverage gate (88.0% -> above the 90%
threshold on the mimi component).

* test: pass host kernel to aarch64-only embeddings conv callsites

pointwise_conv_hwc and standard_conv_hwc_rect gained a leading
emel::kernel::sm parameter, but the aarch64-only #else blocks in
vision_embedding_lane_tests.cpp still used the old signatures; x86_64
hosts preprocess those blocks away, so the break only shows on arm64.

* docs: remove injected atmux tooling block from AGENTS.md

* bench: pin cmake policy floor for sentencepiece subproject

sentencepiece v0.2.0 declares cmake_minimum_required(<3.5); CMake 4
removed that compatibility, so the mimi-suite reference configure fails
on CMake 4 hosts without CMAKE_POLICY_VERSION_MINIMUM.

* scripts: make coverage ctest timeout overridable

ctest's built-in 1500s default kills the O0 coverage-instrumented
generator_and_runtime and text shards on slower hosts; expose
EMEL_COVERAGE_CTEST_TIMEOUT with the same 1500s default.

* docs: regenerate machine architecture docs and benchmark table

generate_docs.sh output for the io/mmap advise flow, kernel dispatch
rows, graph orchestration, and the speech_codec_mimi benchmark entries
added on this branch; the committed generated docs predated them.

* scripts: optional stack raise for coverage ctest runs

Apple clang's O0-instrumented frames for the deep SML machine paths
overflow the default 8MB main-thread stack; the SIGSEGV lands in
doctest's crash handler, which wedges under gcov so ctest reports a
timeout instead of a crash. EMEL_COVERAGE_STACK_KB raises the limit for
the ctest subshell; default behavior unchanged.

* sm: bounded idle wait in thread_pool_scheduler worker loop

Under gcov-instrumented builds on macOS the coverage shards wedged at
fixture teardown: stop_workers() released a stop permit per worker,
seven of eight workers exited, and the last slept in
counting_semaphore::acquire()/__ulock_wait forever while join() waited
on it. Permit accounting is sound (a consumed stop permit
synchronizes-with its release and makes stopping_ visible), so the hang
is a missed wakeup below the scheduler. Replace the bare acquire on the
idle path with try_acquire_for(1ms) plus a stop re-check: a missed
notify now costs at most 1ms instead of a deadlock, and the warm
spin-claim fast path is untouched.

* test: cover io/mmap advise failure and unsupported-platform routes

The unsupported-platform row is unreachable through dispatch on
supported platforms (its guard sits after the supported row), and the
attempt-failed route needs a platform madvise rejection macOS does not
produce for in-mapping windows; exercise the pure guard predicates and
marking effects directly, the processor_action_branch_tests pattern.

* bench: add personaplex-mlx Mimi reference lane

Second mimi_compare/v1 reference backend: the pinned personaplex-mlx
checkout's MLX Mimi codec (tools/bench/personaplex_mlx_ref.txt) driven
per-frame by tools/bench/speech/personaplex_mlx_mimi_driver.py inside
its own venv, mirroring moshi_reference_driver's CLI so
tools/bench/mimi_compare.py consumes either lane unchanged. Both lanes
share Kyutai e351c8d8-checkpoint125 weight provenance (sha256-pinned
safetensors for MLX, the converted GGUF for EMEL); the driver builds the
model at the checkpoint's 32 codebooks and slices the requested n_q,
which residual VQ makes exact.

Deep RVQ layers quantize near-zero residuals, so their codes cascade
apart between any two float implementations (440 Hz tone surface: EMEL
vs MLX 0.52 full-code match, decode PSNR 21.8 dB, while Kyutai's own
rustymimi vs MLX is 0.66). The lane therefore gates the numerically
stable early streams (new --prefix-streams/--min-prefix-match in
mimi_compare.py: first 4 streams >= 0.90, measured 0.96) with a 0.40
full-match sanity floor; token exactness remains the moshi_cpp lane's
gate. Manifest records added for the lane files, both reference-backend
descriptors, and the manifest registry itself (previously unmatched
tools/bench paths escalated suite-less gates to full).

* bench: speech_codec_mimi_mlx timing suite with personaplex-mlx reference

Measurement-only per-frame timing on the real Mimi artifact: both lanes
self-report steady-state loop time over a 10 s signal (model load
excluded) — mimi_emel_parity_runner gains --timing, both reference
drivers print encode/decode_ms_per_frame to stderr, and
mimi_compare.py --timing-audio collects them into the report JSON plus
bench-style compare rows. Wired as gate suite speech_codec_mimi_mlx
(whisper_compare pattern), host-gated to Apple Silicon macOS.

Measured on this M-series host: EMEL aarch64 CPU lane 183.3 ms/frame
encode / 139.2 ms/frame decode vs personaplex-mlx (Metal) 5.1 / 4.0
ms/frame — the arm64 lane misses the 80 ms real-time budget the x86
lane meets at ~65 ms/frame; quantifies the aarch64 performance-port gap
alongside the existing exactness-port follow-up.

* review: script and docs fixes from PR #90 bot feedback

- tokenizer_tests: drop stray double semicolons after make_bpe_vocab().
- build_jobs.sh: floor the computed job count at one instead of forcing two,
  so memory-constrained hosts keep the protection the clamp exists for.
- bench_mimi_compare.sh: source the shared build-parallelism clamp and pass
  the computed job count to the compare-tools build.
- setup_personaplex_mlx_reference.sh: run the GGUF converter with the
  discovered interpreter so hosts without a python3 shim work end to end.
- AGENTS.md: describe the actual build_jobs.sh contract (6GB default per-job
  budget, EMEL_BUILD_JOB_MEM_GB override) instead of the stale 4GB figure.

* io/mmap: reject unknown advise kinds explicitly

An out-of-range advice value (for example a cast at a C ABI boundary)
matched none of the three kind guards, so the completion chain stopped in
state_advise_kind_decision with no error published and no return to ready.
The kind decision now has a guard-level catch-all routing to a dedicated
invalid-kind error decision (new error::invalid_advise_kind), mirroring the
invalid-range route, and the machine finishes the dispatch. Reproduced first
with a failing route test; a well-formed advise after the rejected one
succeeds.

* mimi: validate model contract bounds and surface unexpected-event errors

Contract validation (the dry-run bind walk) now rejects, with bind_failed:
- tensors whose metadata is present but whose weight storage was never
  bound (tensor_has_storage joins the float acceptance predicate),
- transformer layer counts beyond the fixed k_max_transformer_layers array,
- RVQ splits beyond the fixed k_max_quantizer_levels arrays,
- head sizes the fp16 rotary halving cannot serve (odd dim/heads), which
  previously initialized successfully and then silently skipped the
  transformer block at compute time (reproduced first with a failing test).

External events arriving in states that do not model them (initialize while
session_ready, reset_stream while uninitialized) previously took the
unexpected-event no-op and reported success. reset_stream now dispatches
through the same request/ctx runtime-event pattern as the other public
events, and effect_on_unexpected records the new error::unexpected_event
(propagated to error_out when the request carries one), so those dispatches
fail loudly.

* docs: regenerate io/mmap and mimi machine docs

* tensor/window: harden bind/acquire/unbind contracts from PR #90 bot review

Bind chain:
- extents are validated against the mapped source size (byte_size > 0 and
  offset+size within file_size_bytes, overflow-safe), so a malformed extent
  can no longer make the I/O worker read past the mmap.
- slot/prefetch sizing moved to a streaming-branch guard
  (guard_bind_streaming_config_valid): passthrough binds accept the
  documented zero slot fields instead of reporting invalid_request, and an
  invalid streaming config reports invalid_request explicitly.
- slot allocation is a nothrow attempt routed by explicit alloc-decision
  guards (new state_bind_alloc_decision, error::slot_alloc_failed) instead
  of throwing operator new inside a noexcept action.
- every post-map rejection (budget_too_small, invalid streaming config,
  alloc failure) releases the source mapping and resets the window, so
  repeated failing binds no longer exhaust the shared io_mmap pool.
- bind done publication is guarded on the optional on_done callback for
  both the passthrough and streaming routes.

Acquire chain:
- a slot committed as failed by a background prefetch now resubmits its
  load (retry encoded in the transition graph; a second failure still
  publishes slot_copy_failed) instead of matching no settle guard and
  stranding the actor.
- the unbound publish route honors the optional on_error callback.

Unbind chain:
- the release is an attempt whose outcome routes the finish decision; the
  window resets only after a successful release, and the failed-release
  exits return to the still-bound mode (ready/passthrough_ready) so the
  reserved handle can be retried, instead of forgetting the mapping.
- the unbound finish route honors the optional on_error callback.

Ownership:
- the machine destructor runs the modeled unbind drain, joining in-flight
  slot loads before slot storage is freed.
- copying the context no longer shallow-copies owned slot storage: copies
  carry the collaborators and a fresh window (the sanctioned construction
  copy is pre-bind), and assignment is deleted.

* text/generator: streamed decode rows for the preselected argmax family

run_decode_preselected_argmax already threads window_mode through run_layer;
this exposes it: run_kernel_scalar_preselected_argmax_mode gains the wmode
parameter, and the four serial flash preselected routes (q8_k,
native_quantized_q8_k, native_quantized_kernel, kernel) get streamed
variants as explicit guard-routed rows sitting above their resident and
parallel siblings (guard_decode_stream_window_ready composition, the same
convention as the materialized streamed rows). Previously an active tensor
window on a preselected-argmax session silently decoded from the
mmap-resident records. Proven by a new engagement test: streaming model B's
bytes under a generator holding model A's records makes the preselected
decode follow the window bytes.

* bench: document that the weight-streaming window lane streams via the
preselected streamed rows

The emel_window lane keeps selection_mode::preselected_argmax; with the
preselected streamed decode rows in place an active tensor window routes
the serial slot-consuming variants, so the lane now measures actual
streamed decode instead of silently reading mmap-resident records.

* docs: regenerate tensor window and text generator machine docs

* style: clang-format the review-batch hunks

Keeps the audited wrapper region's spaced pointer style (int32_t * err_out):
the generator route-choice audit scans that region for *err_out dereference
writes, and clang-format's pointer binding would make every parameter
declaration match the pattern.

* test: cover the unbind present-callback composes and failed-guard range check

* mimi: route the unexpected-event error channel through guarded transitions

effect_on_unexpected chose whether to write error_out with a runtime null
check inside the action. Every unexpected-event row is now a guarded pair -
[guard_unexpected_error_out_present] / effect_mark_unexpected_and_store and
[guard_unexpected_error_out_absent] / effect_mark_unexpected - mirroring the
guard_has_error_out convention on the modeled routes, so the channel choice
lives in sm.hpp transitions and both effects are straight-line.

* tensor/window: guard the already-bound bind error publication

A rebind while bound called effect_publish_bind_error unconditionally, so a
retry without an error callback dereferenced an empty emel::callback. The
already-bound rows are split by guard_bind_error_callback_present/absent;
the no-callback route records the error and reports failure via the return
value.

* review: kernel/converter/bench-lane fixes from PR #90 round-2 feedback

- kernel: can_run_get_rows requires bound index storage before the
  validation scan dereferences src1 (new null-index rejection test).
- moshi_gguf_convert: the CRC unmangle table detects collisions and raises
  with both canonical names instead of silently restoring the wrong one.
- bench: new EMEL_BENCH_SKIP_MOSHI_REFERENCE option; the personaplex-mlx
  lane passes it ON (and moshi_cpp passes OFF explicitly) so the MLX-only
  compare never fetches moshi.cpp/SentencePiece.
- setup scripts: converter side inputs (--config/--tokenizer) are freshness
  prerequisites of the enriched GGUF, so refreshed metadata forces a
  reconvert on both reference lanes.

* revert: drop quality-gate timing snapshot churn

Snapshot files are not updated without explicit consent; restore
snapshots/quality_gates/timing.txt to its reviewed state (925ae1d7). The
gate rewrites it locally on every run; those rewrites stay uncommitted.

* tensor/window: owner-provided slot storage; modeled submit-rejection outcome

The machine no longer allocates during dispatch: bind_window_request carries
a caller-provided, k_slot_alignment_bytes-aligned slot arena that the
streaming activation partitions into slots (pure pointer bookkeeping),
validated by guard_bind_slot_storage_sufficient at the budget decision. An
undersized arena routes the new error::slot_storage_too_small explicitly
(and releases the mapping); state_bind_alloc_decision, the nothrow
allocation attempt, and error::slot_alloc_failed are deleted. reset_window
clears slot views without freeing - the caller owns the arena for the
lifetime of the bind, and unbind (or the destructor drain) joins in-flight
loads before the arena may be released.

bind_and_submit_layer_load no longer selects an inline execution fallback
when the pool rejects a submit: rejection (unreachable within the machine's
own invariants, at most slot_count <= 8 loads in flight against a 128-entry
queue) now fires the un-run ticket, committing the slot as failed through
the normal completion route where the acquire chain's failed-slot resubmit
and slot_copy_failed rows make the outcome explicit.

The alloc-failure functor checks are replaced by a machine-driven
undersized-arena test; fixtures, the stream-window generator rig, and the
weight-streaming bench rig allocate their arenas owner-side before any
dispatch.

* text/generator: typed stream-acquire failure code through the compute-error route

A streamed decode whose per-layer window acquire fails now writes the new
k_error_stream_acquire through the run_kernel_fn err_out channel (threaded
run_kernel_scalar_mode -> run_decode -> run_layer and the preselected
equivalents), so the failure is attributable to the tensor-window
collaborator at the modeled compute-error routing layer instead of folding
into a generic kernel false. Covered by a machine-driven test: a generator
wired stream_active against an unbound window fails generation cleanly.

* kernel: guard zero KV heads before the flash-attn modulo

can_run_flash_attn_ext computed head_count % kv_head_count in shape_match's
initializer regardless of the separate dims_present zero check, so a
malformed request with zero KV heads hit undefined behavior inside the
guard. The modulo is now short-circuited on kv_head_count != 0 in the same
expression. Reproduced with a zero-KV-heads rejection test on both lanes.

* tensor/window: route bind-rejection release outcomes explicitly

effect_release_rejected_source ignored a failed release_mapping and reset
the window anyway, discarding the only handle to a slot the mmap actor
keeps reserved for retry. The release is now an attempt whose recorded
outcome routes the bind error exits: a successful release resets to
unbound (unchanged), a failed release retains the source in a
passthrough-like bound state so unbind_window can retry it (mirroring the
unbind failure exits). Pre-map rejections default to the plain unbound
exit.

* bench: skip the llama.cpp fetch for the MLX-only Mimi compare

The EMEL_BENCH_NEEDS_LLAMA clause fetched llama.cpp/ggml for
speech_codec_mimi even when EMEL_BENCH_SKIP_MOSHI_REFERENCE builds no moshi
reference driver; the MLX lane now skips both fetches.

* generator/mimi: streamed q8_k-logits decode row; reject short transposed kernels

- The materialized native_quantized_q8_k_logits decode rows sat above the
  streamed rows, so q8_k materialized decode with an active tensor window
  silently read resident records; the route gains its streamed variant
  above the resident/parallel siblings (same recipe as the other streamed
  rows).
- The mimi plan walk rejects transposed-conv kernels shorter than their
  stride (seanet layers and the final upsample): the decode overlap-add
  carries taps - stride samples of state, which a shorter kernel would
  index outside. Reproduced with a mutated-metadata bind test.

* test: machine-driven coverage for no-callback rejection exits and the seanet taps check

* fix(kernel): reject metadata-only operands in rope/im2col/conv_transpose guards

The exec kernels read through src0/src1 storage; a metadata-only tensor
(null data) previously passed can_run_* and dereferenced null inside the
action. Add storage checks to the guards and machine-driven rejection
tests.

* fix(generator): route packed_q8_0/q8_k flash decode through streamed variants when a tensor window is active

The resident parallel rows for packed_q8_0 and q8_k matched ahead of any
window-aware route, so an active streamed tensor window was ignored for
those dtype families. Add streamed serial decode rows above the
resident/parallel siblings (SML first-match) with guards composing
guard_decode_stream_window_ready with the existing dtype-readiness
predicates, plus the wrapper/action/CPO plumbing.

* fix(generator): restore pristine weight views after streamed decode

bind_streamed_block_views repoints each block's matmul views at the shared
per-role stream records; nothing restored them, so a later resident step
(the next generate_run's prefill) read the last acquired slot's clone
instead of the per-layer tensors. Reproduced by the new
restores-resident-weight-views test (second mixed run produced
worldworld), fixed by reset_stream_block_views after the streamed layer
loop on both decode drivers, including the failure path.

* fix(tensor/window): retry the release once in the destructor drain

A failed release keeps the machine on the modeled retained-bound row for
caller retry, but the destructor dispatched unbind only once and then
context destruction discarded the retained handle, permanently
quarantining the mmap actor's slot even for transient unmap failures. A
second bounded dispatch retries through the existing retained-bound row;
a persistent failure still lands in the documented mmap quarantine.

* fix(scripts): pass mimi compare build targets as one explicit array

The empty-array plus-expansion actually yields zero words (verified on
bash 3.2 and 5), but an always-non-empty build_targets array removes the
ambiguity the reviewer flagged.

* test(generator): cover the preselected streamed failure path

Adds the preselected twin of the window-cannot-serve rejection test and
drops the defensive min-ternary in reset_stream_block_views (pristine is
empty or exactly blocks.size() by construction), closing the changed-line
coverage gap on the restore fix.

* fix(mimi): reject zero transformer context, short streaming conv kernels, and empty rvq hparams at bind

compute_transformer takes position modulo the context, so a zero
transformer_context divided by zero on the first frame instead of failing
bind. Non-transposed streaming convs (seanet strided layers and the
downsample conv) accepted taps < stride, wrapping the taps - stride state
span negative in the sizing accounting. codebook_dim/card are now rejected
explicitly as hparams: the pinned fixture's metadata cross-check already
caught zero values, but a synthetic model with matching zero-width tensors
would have initialized with an unbound quantizer child. Reproduced the
zero-context and both conv cases with failing bind-rejection subcases
first.

* fix(generator): stream the nonflash decode routes through an active tensor window

With a bound window and decode_flash_runtime_supported false, the nonflash
decode blocks had no streamed variants, so the resident rows won and the
decode read mmap-resident records instead of acquiring layer slots.
Reproduced first with kv-width-mismatch engagement tests (mixed run
produced hellohello instead of helloworld on both selection modes). Adds
streamed serial variants for the five nonflash decode families and the
four nonflash preselected families, routed by the existing streamed guards
above their resident siblings.

* fix(mimi): validate frame rate, rotary max period, and tensor byte extents at bind

frame_samples was cast from sample_rate / frame_rate with no finiteness or
range check (undefined for NaN or out-of-range ratios; rejection previously
depended on platform cast behavior downstream). transformer_max_period <= 0
reached log() in compute_transformer. tensor_has_storage accepted any
non-empty buffer, so matching element metadata with data_size == 1 read
past its storage during the prepare copies (the q8_0/f16 raw prepares had
the same gap). Reproduced the max-period and undersized-storage cases with
failing bind-rejection subcases first; the frame-rate cases were rejected
on this host only through UB casts and now reject well-definedly at plan
time.

* fix(kernel): require flash-attn operand storage and finite rope frequencies

can_run_flash_attn_ext accepted metadata-only Q/K/dst operands; a null key
cache segfaulted in the exec's K row scan (reproduced: the new null-K case
crashed pre-fix). can_run_rope accepted infinite freq_base/freq_scale via
bare positivity checks and reported success while computing NaNs
(reproduced with CHECK_FALSE failures pre-fix). Both guards now reject.

* fix(generator): forward the compute error pointer through the preselected decode wrapper

run_kernel_scalar_preselected_argmax_mode discarded the graph processor's
err_out, so a streamed direct-preselected decode whose window acquire
failed would write the stream-acquire error through a null default pointer
in run_layer. The decode branch now forwards err_out, keeping run_layer's
always-valid-pointer contract true. The route is only reachable via the
direct preselected q8 families (host-unexercisable here, same LFM2
disposition); the window-cannot-serve preselected test now uses no
samplers so it exercises the streamed decode path rather than the
invalid-request row.

* fix(scripts): discover the python interpreter in the mimi compare wrapper

Hosts with python3.12/python3.13 but no python3 shim (the environment the
MLX setup script supports) failed the tool check and the later WAV
generation and comparison steps. The wrapper now resolves PYTHON_BIN
(env override, then python3, python3.13, python3.12) and uses it for
every python invocation.

* fix(kernel): accept bf16 sources in the get_rows dispatch validation

has_required_src0's generic fallback omitted bf16 from the supported-dtype
list, so bf16 get_rows requests were rejected as invalid before the
explicit bf16 conversion row could match (reproduced: the new bf16 gather
case failed pre-fix). The op_get_rows branch now accepts bf16 sources with
the same layout and element checks as the other float dtypes.

* fix(mimi): bound tensor rank before scanning dimensions

tensor_elements iterated n_dims over the four-wide dims array, so a
malformed rank read past the record before any shape check ran; rejection
depended on the garbage product mismatching the expected count. Ranks
outside 1..4 now count as zero elements, which every validation caller
rejects, and the bind-rejection subcase pins it.

* fix(moshi): validate every acoustic rvq codebook in the mimi contract

The quantizer contract probed only the last rvq_rest level, so a model
missing an intermediate codebook validated successfully and then failed at
bind_rvq_split during initialization (reproduced: hiding level 1 of the
tiny fixture passed the contract pre-fix). The check now walks every
expected level and requires at least one acoustic level.

* fix(moshi): require the codebook shape for every acoustic rvq level

The per-level probe checked existence only, so an intermediate rvq_rest
codebook with the wrong {codebook_dim, card} shape validated at contract
time and failed later in bind_rvq_split (reproduced: the new wrong-shape
subcase passed validation pre-fix). Each level now goes through
require_tensor_shape with the expected embedding shape.

* fix(generator): route flash streamed preselected actions through the preselected wrapper

The four flash streamed preselected_argmax actions called the generic
request_phase_compute, which leaves io.selected_token_out and
selected_score_out null; the preselected mode wrapper dereferences both,
so an active window on a flash direct-preselected route would crash
instead of returning the selected token. They now use
request_phase_compute_preselected_argmax like their resident siblings and
the nonflash streamed rows. The route is only reachable via the direct
preselected q8 families (host-unexercisable here, same standing LFM2
disposition).

* fix(kernel): cap convolution guard extents before signed length arithmetic

can_run_im2col and can_run_conv_transpose_1d cast public uint64 extents to
int64 and evaluated the output-length formulas before shape rejection, so
extents near INT64_MAX hit signed-overflow UB in the guard (rejection
previously depended on the overflowed garbage mismatching dst dims). With
every operand capped at 2^31 and int32 op params, the largest intermediate
stays under 2^63; oversized extents now reject explicitly and the new
huge-extent cases pin both ops.

* fix(bench): bound the mimi bench tensor span before binding

The probe count was assigned unchecked; a fixture with tensor_count past
the fixed tensors array would let the loader write past it. Mirrors the
parity runner's guard.

* fix(bench): fail mimi decode comparison on pcm length mismatches

min-prefix comparison could hide a truncated or extended decode stream
behind the encode gates. Both lanes verified to produce equal-length PCM
on the pinned tone (moshi_cpp and personaplex-mlx --run-only).

* fix(bench): validate every configured block and codebook level at conversion

The converter probed only the last LM transformer layer, the last
depformer layer, and a bare acoustic-codebook count, so a source GGUF
missing an intermediate level (or carrying a wrong-shaped codebook) still
converted into an artifact the C++ contract rejects. Each configured
transformer/depformer layer and each expected rvq_rest level (with the
{codebook_dim, card} shape) is now required at conversion time.

* fix(bench): route personaplex-mlx manifest records to the mlx suite

MLX-only inputs (driver, pin, backend descriptor) selected the generic
speech_codec_mimi suite, which runs the moshi.cpp compare lane; scoped
gates therefore never exercised the MLX lane on MLX-affecting changes.
The records and the pinned manifest now select speech_codec_mimi_mlx.

* fix(mimi): require storage for arena and frame spans in the request guards

The arena-capacity and frame-request guards accepted sized spans with null
data(): a null arena crashed in the bind walk or reset_streaming_state,
and null pcm/codes spans dereferenced null inside the compute actions.
Reproduced all three with failing machine-driven cases first (null
prepared arena, null encode pcm, null decode output); the guards now
require non-null storage and route the existing arena_capacity and
request_shape errors.

* fix(tensor/window): reject null slot storage before partitioning

A null arena base is 64-byte aligned and can carry any span length, so the
storage-sufficient guard accepted it and the slot partition derived every
worker destination from address zero (reproduced: the null-storage bind
was accepted pre-fix and corrupted the follow-up bind). The guard now
requires a non-null base and the reject routes the existing
slot_storage_too_small row; the new lifecycle test pins it including the
successful retry bind.

* fix(bench): bound the mimi bench name arena before copying

Tensor names from a regenerated fixture whose aggregate length exceeds the
fixed name_storage wrote past it during fixture setup. Mirrors the parity
runner's bound; setup now fails cleanly.

* feat(io): constructor-injected platform boundary for mmap and staged_read

Per the review decision on the window functor-check thread (option c):
the io/mmap actor's OS mapping calls move behind a platform_ops struct
injected through the context at construction (production defaults via
default_platform_ops(); the ops select no behavior - every route stays an
explicit guard/transition), and the staged_read platform capability
becomes a construction-time context field defaulting to the compile-time
probe. Both io sms inherit the context DI constructor.

This makes the previously unforceable failure routes machine-drivable:
the window functor-check test is replaced by four process_event-driven
tests covering the failed-release retained exits in both modes (streaming
and passthrough, with acquire proving the bind stays usable and the retry
unbind releasing), the destructor drain's transient-release retry (unmap
attempted exactly twice), and the failed-slot resubmit chain publishing
slot_copy_failed. The STG-07 assert now checks the actual invariant (no
dispatch-local payload; trivially copyable, one capability flag wide).

* fix(tensor/window): reject null plan spans before validation scans

bind_window accepted sized-but-null layer_weight_counts/extents spans and
scanned them during validation before the invalid-request path could run
(reproduced: the null-plan-span bind crashed validation pre-fix). The
request guard now requires storage for both plan spans and routes
invalid_request.

* fix(moshi): require dtype-sized tensor payloads in contract checks

tensor_has_storage counted any positive data_size as storage-backed, so
exact-shape tensors over a one-byte buffer satisfied the LM/Mimi/voice
contracts and failed later at runtime binding (reproduced: the new
data_size=1 subcase passed the contract pre-fix). The check now requires
the dtype-implied byte count (f32/f16/bf16 widths, q8_0 blocks, a one-byte
floor for dtypes the families never carry).

* fix(mimi): require storage for directly driven child request spans

The encoder/decoder/quantizer children are independently usable actors,
but their shape and capacity guards accepted sized spans with null
storage (reproduced: null encode pcm and null decode output reached the
compute effects pre-fix). All child span guards now require non-null
storage and route the existing request_shape/buffer_capacity errors.

* fix(bench): complete the mimi compare suite manifest coverage

The shared compare wrapper and comparator carry MLX-specific branches, so
they now map to speech_codec_mimi_mlx as well; both reference setup
scripts (moshi.cpp and personaplex-mlx) gain records so a setup-only
change selects its maintained gate instead of full-gating or skipping.

* fix(moshi): harden lm delay bound, frame-rate finiteness, and semantic rvq levels

lm.n_q + 1 evaluated a signed addition before the k_max_delays bound could
reject, overflowing on a malformed n_q (rejection previously depended on
the wrapped value mismatching the delay count); the bound is now
n_q < k_max_delays with no addition. The mimi hparam gate accepted NaN and
infinite frame rates (NaN compares false against <= 0; reproduced with the
new kv-backed subcases), and the quantizer contract probed only the first
semantic codebook while bind_rvq_split consumes levels 0..semantic_n_q-1
(reproduced: declaring a second semantic level the fixture lacks passed
validation). Every semantic level now gets the full shape probe, which
also rejects splits whose codebooks cannot all be present.

* fix(io/mmap): seed missing platform ops from the defaults at construction

A partially populated injected platform_ops left null function pointers
that the effects would call through (reproduced: a context built from an
unmap-only ops struct carried null open/map/close pre-fix). The context
constructor now seeds every null field from default_platform_ops(), so an
injector overriding one op keeps production behavior for the rest.

* fix(moshi): reject odd mimi attention head sizes at the hparam gate

dim divisible by tr…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants