feat(abi): MiniMax-H3 video+audio generation through the ONE surface — vllm_video_*, ABI v12, /v1/videos via the library seam, thin-client examples

.agents/NOW.md

@@ -28,7 +28,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint
 | Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | — |
 | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning |
 | `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable |
-| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 1 LANDED (#121)**: Parakeet ASR on the surface (`vllm_transcribe` v11, `/v1/audio/transcriptions`, ratchet 12→11) | Next fold row |
+| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 2 LANDED (#123)**: H3 video on the surface (`vllm_video_*` v12, `/v1/videos` via the seam, both examples thin clients, ratchet 11→9); ROW 1 (#121) before it | GB10 re-verify residual; next fold row |
 
 In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna
 bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.

.agents/specs/one-surface-abi.md

@@ -1,6 +1,6 @@
 # ONE SURFACE — every capability ships through the C ABI
 
-Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11.**
+Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign).**
 
 ## The defect
 
@@ -28,7 +28,7 @@ stay OPEN. The complete, code-grounded matrix (all 30 archs + every off-registry
 
 | capability | registered? | why still off-surface | only real path via | example size |
 |---|---|---|---|---|
-| MiniMax-H3 video+audio gen | NO | no arch, no video C-ABI; served via example-injected `VideoRunner` | `examples/minimax_h3_gen`, `examples/server` `/v1/videos` | 1293 lines |
+| MiniMax-H3 video+audio gen | NO (diffusion lane) | **CLOSED (ROW 2)** — `vllm_video_*` on ABI v12; `/v1/videos` drives the library seam | library seam `MiniMaxH3VideoEngine`; examples are thin clients | 216 lines (was 1293) |
 | Laguna | YES | keep-quant/NVFP4 decode example-only; registry forward `VT_CHECK`s non-bf16; GGUF dispatch unreachable | `examples/laguna_gen` | 415 lines |
 | Kimi-Linear | YES | registry leg is a stateless recompute reference; §18/§19 incremental entry points are private | `examples/kimi_linear_gen` | 318 lines |
 | DeepSeek-V4 | YES | KV spec is a "never exercised" stub; registry forward discards attn_meta/kv | `examples/deepseek_v4_gen` | 240 lines |
@@ -71,11 +71,39 @@ behaviour.
 4. `examples/server` routes through the SAME entry point, so HTTP and FFI cannot
    drift.
 
-## Proposed ABI shape for video (first slice)
+## Proposed ABI shape for video (first slice) — SHIPPED (ROW 2, ABI v12)
 
 Mirrors the existing engine idiom: an opaque handle, a params struct with a
 `_default()`, an explicit free, `vllm_last_error` for diagnosis.
 
+**As-shipped deltas from the proposal below (each argued during ROW 2,
+2026-08-08; `include/vllm.h` is the binding text):**
+- `vllm_video_model_params` GAINS `prompt_embeds_path` (without an encoder
+  there is NO conditioning path — both pre-fold consumers had this arm),
+  `partition` (the #77 guard refuses every full render without a declared
+  partition, so omitting it would make the ABI unable to render at all) and
+  `fp4_resident` (the gated NVFP4 Marlin arm); `encoder_max_layers` stays a
+  C++-seam knob.
+- `vllm_video_params` GAINS `output_dir` (the result needs a destination) and
+  DROPS `duration_seconds`/`task` (derivable: `num_frames` expresses duration,
+  and the task is resolved from the references exactly as upstream
+  `_resolve_task` does; both remain on the C++ seam for the server).
+- `vllm_video_result.mux_argv` is `char**`, NULL-terminated (execvp-ready),
+  freed via `vllm_video_result_free`.
+- ADDED `vllm_video_mux_params(_default)` + `vllm_video_mux_argv(_free)`: the
+  engine-free composer `minimax-h3-mux` needs to be a `vllm.h` client (W4);
+  the encoding contract stays the library's, the caller execs.
+- NAMED RESIDUALS of the first slice: ONE `ref_image` (multi-image ref2va is
+  C++-seam-reachable only), and the pre-fold example's diagnostic modes
+  (`--denoise-only`, `--dump-params`, `--encoder-only`/`--save-embeds`,
+  `--decode-latent`, `--roundtrip`, `--prompt-image`, `--cond-image`,
+  `--dry-run`) were deleted with the private pipeline — the capabilities they
+  probed are gated by `test_minimax_h3`/`test_minimax_h3_video_fold`, and the
+  GB10 speed recipe moves to the seam/ABI (re-verification residual below).
+- The pre-fold CPU host-f32 GGUF arm (the example default with NEITHER
+  `--keep-quant` nor `--dequant-bf16`) is not on the ABI: `dequant_bf16=0` is
+  keep-quant (the gated arm the fold goldens were captured on).
+
 ```c
 typedef struct vllm_video_engine vllm_video_engine;
 
@@ -140,11 +168,21 @@ embedder produce an MP4 without reinventing the command: the library composes it
 
 ## NOT claimed
 
-The VIDEO ABI shape above is a proposal, not an implementation, and it has not
-been reviewed against an embedder other than LocalAI. The ONE landed slice is
-audio transcription (ROW 1, 2026-08-07): `vllm_transcribe` + params/result
-structs on ABI v11, the `ParakeetTranscriber` library seam, task-conditional
-`/v1/audio/transcriptions`, registry refuse-by-task, and the example as a thin
-`vllm.h` client — gated byte-identical to the pre-fold transcripts. Video,
-Laguna/DeepSeek/Kimi fast decode, embeddings and multimodal input remain open
-rows of this program.
+The video ABI has not been reviewed against an embedder other than LocalAI.
+TWO slices are landed: audio transcription (ROW 1, 2026-08-07 — ABI v11
+`vllm_transcribe`, the `ParakeetTranscriber` seam, task-conditional
+`/v1/audio/transcriptions`, example as a thin client, byte-identical
+transcripts) and video+audio generation (ROW 2, 2026-08-08 — ABI v12
+`vllm_video_*`, the `MiniMaxH3VideoEngine` seam, `/v1/videos` through it, both
+H3 examples as thin clients, frames+WAV byte-identical to the pre-fold binary
+on the committed fold fixture; three-arm gate `test_minimax_h3_video_fold` +
+the v12 `test_capi` section). ROW 2 residuals, named: (1) GB10 real-video
+re-verification (real checkpoints through the v12 ABI + the folded server;
+the box is running the Kimi campaign — CPU fold gates are the landed
+evidence); (2) DISCLOSED server-path numeric deltas (no goldens existed): the
+pre-fold `/v1/videos` runner drew single-stream legacy-uniform noise and
+defaulted to the host-f32 GGUF arm — it now shares the ratified recipe
+(dual-stream splitmix64 Gaussian, keep-quant default), which kills the exact
+HTTP-vs-CLI drift this row exists to prevent; (3) multi-image ref2va on the
+ABI. Laguna/DeepSeek/Kimi fast decode, embeddings and multimodal input remain
+open rows of this program.

.agents/specs/surface-coverage-2026-08-07.md

@@ -24,7 +24,7 @@ The four surfaces, and the public boundary the guard draws:
 
 | Rank | Gap | Registered | Servable | C-ABI | Where the capability actually lives |
 |---|---|---|---|---|---|
-| 1 | **MiniMax-H3 video+audio generation** | NO (off-registry diffusion) | `/v1/videos` ONLY via an example-injected `VideoRunner` | NO | `examples/minimax_h3_gen` + `examples/minimax_h3_mux` (whole pipeline) |
+| 1 | **MiniMax-H3 video+audio generation** | **CLOSED (ROW 2)**: still off-registry (a diffusion lane, not a text arch) but library-owned end to end | **`/v1/videos` through the library seam** (`MiniMaxH3VideoEngine` + `MiniMaxH3VideoGenParamsFromRequest`; the server keeps only flag plumbing + the ffmpeg exec) | **`vllm_video_*` (ABI v12)** | library seam `vllm::multimodal::MiniMaxH3VideoEngine`; both examples are clean ABI clients |
 | 2 | **Laguna fast decode** | yes, but forward is a stub | no (stub `VT_CHECK`s non-bf16) | no | `examples/laguna_gen` (keep-quant GGUF + NVFP4 W4A4) |
 | 3 | **DeepSeek-V4 fast decode** | yes, but forward is a W3 stub | no (stub) | no | `examples/deepseek_v4_gen` (keep-quant GGUF) |
 | 4 | **Audio transcription** | **CLOSED (ROW 1)**: Parakeet CTC/RNNT/TDT registered (transcription-only; Whisper/Voxtral still off-registry) | **live `/v1/audio/transcriptions`** (task-conditional; the run_batch line stays a residual) | **`vllm_transcribe` (ABI v11)** | library seam `ParakeetTranscriber`; example is a clean ABI client |
@@ -60,7 +60,7 @@ All three drivers run a PRIVATE host-argmax greedy loop, not the on-GPU sampler.
 
 | Lane | Registered | Code (file:line) | Server | C-ABI | Driver |
 |---|---|---|---|---|---|
-| MiniMax-H3 video+audio GEN | NO | `minimax_h3*.cpp` (~22 TUs), vt op `kMiniMaxH3` | `/v1/videos` via `set_video_runner` (example-injected only; `api_server.h:167`, `api_server.cpp:818`; library never sets it) | NO | `examples/minimax_h3_gen`, `examples/minimax_h3_mux` |
+| MiniMax-H3 video+audio GEN | NO (diffusion lane; **ROW 2** made it library-owned without a registry entry) | `minimax_h3*.cpp` (~22 TUs) + the **`minimax_h3_video.cpp` seam** (ROW 2), vt op `kMiniMaxH3` | `/v1/videos` via `set_video_runner`, the runner now a thin exec wrapper over the LIBRARY seam (`MiniMaxH3VideoGenParamsFromRequest` -> `Generate` -> exec argv) | **`vllm_video_engine_load` / `vllm_video_generate` / `vllm_video_result_free` / `vllm_video_mux_argv` (ABI v12)** | `examples/minimax_h3_gen`, `examples/minimax_h3_mux` = thin `vllm.h` clients |
 | Parakeet/FastConformer ASR | **YES (ROW 1)**: ParakeetForCTC/RNNT/TDT, `parakeet_registry.cpp` (SupportsTranscription-only; text paths refuse by task) | `parakeet_transcription.cpp` seam composes encoder/transducer/audio-processor; the example's private `ReadWav16BitMono`/`LoadVocab`/`DecodeIds` are DELETED (`vllm::Tokenizer` now decodes Metaspace split=true) | **`/v1/audio/transcriptions`** (task-conditional) | **`vllm_transcribe` (ABI v11)** | `examples/parakeet_transcribe` = thin `vllm.h` client |
 | Voxtral audio->text | NO (`VoxtralForConditionalGeneration` unregistered) | `voxtral.cpp` (`vllm::multimodal`) | NO (`/v1/audio/transcriptions` = `run_batch.cpp:188` residual) | NO | tests-only reachability |
 | Whisper audio encoder | NO | `whisper_audio.cpp:174` | NO | NO | tests-only callers |
@@ -77,19 +77,21 @@ Public boundary = `#include "vllm.h"` only. 13 example units; `examples/cli` (vl
 links `vllm::shared`, `#include "vllm.h"` only, `cli/main.cpp:16`) is the sole clean ABI
 client. **ROW 1 UPDATE (2026-08-07): `parakeet_transcribe` is the SECOND clean ABI
 client** — the Parakeet fold rewrote it against `vllm.h` + `vllm::shared` only, and the
-ratchet fell 12 -> 11. The remaining 11 reach `include/vllm/**` / `vt/**` and are
-transition-tracked in `scripts/example-abi-allowlist.txt`:
+ratchet fell 12 -> 11. **ROW 2 UPDATE (2026-08-08): `minimax_h3_gen` and
+`minimax_h3_mux` are the THIRD and FOURTH clean ABI clients** (the video fold, ABI
+v12 `vllm_video_*`), and the ratchet fell 11 -> 9. The remaining 9 reach
+`include/vllm/**` / `vt/**` and are transition-tracked in
+`scripts/example-abi-allowlist.txt`:
 
-- Capability drivers: `deepseek_v4_gen`, `laguna_gen`, `kimi_linear_gen`,
-  `minimax_h3_gen`, `minimax_h3_mux`, `server`.
+- Capability drivers: `deepseek_v4_gen`, `laguna_gen`, `kimi_linear_gen`, `server`.
 - Dev/diagnostic (internal-by-nature, folded for consistency): `bench` (via
   `bench_core.h`), `tokenize`, `dump_container`, `dequant_nvfp4`, `quant_gemm_bench`.
 - Out of the gated `examples/` tree: `benchmarks/vulkan_gemm_ab.cpp` (Vulkan A/B harness).
 
 **Policy (developer-directed 2026-08-07): no permanent exemptions.** Every allowlist entry
 — drivers AND dev/diagnostic tools — is a transition-tracker pointing at a fold row; the
 guard fails on any internal include not tracked, and a shrink-only ratchet
-(`MAX_INTERNAL_REACHING`, 11 since ROW 1) means the count can only fall as folds land, never grow to
+(`MAX_INTERNAL_REACHING`, 9 since ROW 2; 11 since ROW 1) means the count can only fall as folds land, never grow to
 admit a new violation. The public header set is DERIVED from the CMake install rules
 (exactly `include/vllm.h` today), not hardcoded. The guard catches BOTH breach vectors: a
 `#include "vllm/..."|"vt/..."|"src/..."` AND a CMake `-I` grant into the internal tree
@@ -128,7 +130,7 @@ lanes are leaves of `ARCH-ONE-SURFACE` (do not open parallel rows).
 
 | # | Fold | Grow ABI (new `vllm.h` surface) | Then rewrite / delete | Effort | Depends on |
 |---|---|---|---|---|---|
-| 1 | Video+audio gen | `vllm_video_generate` + job/status/content entry points (mirror `/v1/videos`; carry the `VideoRunner` internally, not example-injected) | rewrite `minimax_h3_gen`/`server` as clients; delete driver glue | L | H3 loaders; ffmpeg-mux boundary (ratified in `examples/`) |
+| 1 | Video+audio gen | **DONE (ROW 2, 2026-08-08)**: `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` (ABI v12); `/v1/videos` routes through the SAME `MiniMaxH3VideoEngine` seam (job/status/content stay `VideoJobStore`-served; the runner is now a thin exec wrapper the example injects, because the SPAWN stays in examples/ — the ratified ffmpeg boundary) | **DONE**: `minimax_h3_gen` + `minimax_h3_mux` rewritten as `vllm.h` clients (frames+WAV byte-identical to the pre-fold binary on the fold fixture); server driver glue deleted (54 -> 7 H3 refs, all seam type names) | L | H3 loaders; ffmpeg-mux boundary (ratified in `examples/`) |
 | 2 | Laguna fast decode | make the registered `LagunaForCausalLM` forward route the keep-quant/NVFP4 device path (retire the stub); load keep-quant GGUF/NVFP4 dirs through `vllm_engine_load` | rewrite `laguna_gen`; delete `LagunaForwardGguf*` | M | keep-quant load in the engine loader |
 | 3 | DeepSeek-V4 fast decode | same as (2) for `DeepseekV4ForCausalLM`; real MLA paged KV (retire the W3 stub) | rewrite `deepseek_v4_gen`; delete `DeepseekV4ForwardGguf*` | M | MLA paged-KV topology |
 | 4 | Audio transcription | **DONE (ROW 1, 2026-08-07)**: `vllm_transcribe` (ABI v11) + live `/v1/audio/transcriptions`; ParakeetForCTC/RNNT/TDT registered (SupportsTranscription mirror, refuse-by-task) | **DONE**: `parakeet_transcribe` rewritten as a `vllm.h` client (byte-identical transcript goldens); route live, task-conditional | M | encoder→text seam (LANDED: `ParakeetTranscriber`) |

.agents/state.md

@@ -42148,3 +42148,68 @@ synthetic pair + the P4/P6 pretrained evidence. (4) A real Parakeet
 tokenizer.json may carry normalizer fields `FromHfJson` refuses loudly —
 untestable CPU-side without the checkpoint, fails loud not wrong. (5) Whisper/
 Voxtral remain off-registry (fold #9/#10 of the audit).
+
+## 2026-08-08 — ARCH-ONE-SURFACE ROW 2: MiniMax-H3 video+audio generation folded onto the ONE surface (PR #123)
+<!-- state: 2026-08-08T04:30 -->
+
+**What landed (`row/H3-VIDEO-ABI`, task #283; fold order: grow ABI -> rewrite
+examples -> delete parallel impl).**
+- FOLD GATE FIRST: `tests/vllm/models/minimax_h3_video_fold_fixture.h` writes a
+  deterministic tiny checkpoint set (857KB F32 ComfyUI-GGUF DiT at the :3786
+  reduced geometry + on-disk reduced ViT3D/BigVGAN VAEs + prompt embeds); the
+  PRE-fold `minimax-h3-gen` binary at the branch base (fc636c76) rendered it
+  (--partition fl2va --steps 3 --frames 5 --height/width 32, CPU, keep-quant)
+  and its 8 frames + WAV + both `minimax-h3-mux --print-only` argv lines are
+  COMMITTED goldens (`fixtures/minimax_h3_video_fold/`). Determinism proven
+  (two runs, diff clean).
+- W1 seam: `vllm::multimodal::MiniMaxH3VideoEngine` (`minimax_h3_video.{h,cpp}`)
+  absorbs the example's 1293-line assembly driver AND the server's 354-line
+  /v1/videos twin: 4 DiT loader arms (GGUF keep-quant / dequant-bf16
+  host+streamed / bf16-shard stream / NVFP4 bf16+fp4-resident stream), VAE
+  decoder + lazy encoder halves, the H3-Encoder tower staged once (GGUF or
+  bf16 shards), the #77 partition guard, fl2va/ref2va conditioning, the
+  byte-exact splitmix64 Box-Muller noise streams (VT_H3_GAUSSIAN_NOISE
+  honored; seeded requests derive audio via splitmix64), artifact writing +
+  mux argv. Library SPAWNS NOTHING (mkdir -p became std::filesystem; ffmpeg
+  stays caller-side). `MiniMaxH3VideoGenParamsFromRequest` = the ONE
+  /v1/videos mapping.
+- W2 ABI: `vllm_video_engine(_load/_free)`, `vllm_video_model_params(_default)`,
+  `vllm_video_params(_default)`, `vllm_video_generate`, `vllm_video_result(_free)`
+  + `vllm_video_mux_argv((_params_default)/_free)`; VLLM_ABI_VERSION 11 -> 12,
+  test_capi floor >= 12. As-shipped deltas vs the ratified proposal argued in
+  the spec (+prompt_embeds_path/+partition/+fp4_resident/+output_dir/+mux
+  composer; -task/-duration; ONE ref_image). Refuse-both-directions pinned:
+  video-load on a text dir names vllm_engine_load; vllm_engine_load on the H3
+  dir fails byte-for-byte as captured at v11.
+- W3 server: the VideoState block is DELETED; /v1/videos = seam Load + a
+  runner lambda (FromRequest -> Generate -> fork/execvp(mux_argv)). Direct
+  MiniMaxH3 refs 54 -> 7 (all seam type names). DISCLOSED numeric deltas on
+  the server arm (no goldens existed; the drift WAS the defect): legacy
+  single-stream uniform noise -> the ratified shared recipe; host-f32 GGUF
+  default -> keep-quant; new `--video-dequant-bf16`.
+- W4 examples: `minimax_h3_gen` (1293 -> 216 lines) + `minimax_h3_mux` are
+  `vllm.h`+`vllm::shared` thin clients; BOTH byte-identical to the pre-fold
+  binaries on the fixture (frames+WAV cmp clean; mux argv diff clean).
+  Ratchet: both allowlist rows removed, `MAX_INTERNAL_REACHING` 11 -> 9 +
+  equality pin + spec claims moved; abi-capability video row closed; FEATURES
+  video row -> reachable naming the four symbols.
+
+**Gates.** 3-arm fold gate `test_minimax_h3_video_fold` 4/131 GREEN (seam ==
+replicated pre-fold pipeline == committed goldens, byte-identical; guard
+refusals; FromRequest field-complete). test_capi 40/373 incl. the v12 golden
+e2e THROUGH the C marshalling; vllm_capi_c_check strict-C11 green.
+test_minimax_h3 75/75 unchanged. test_openai_api_server 45/566 incl. the
+socket-level "routes do not exist without a runner" 404 pin; test_video_api
+14/14. check-surface-coverage green with the SHRINK enforced + its 46-test
+mutation suite green.
+
+**Residuals (honest).** (1) GB10 real-video re-verification through the v12
+ABI + folded server (real checkpoints; box on the Kimi campaign — CPU fold
+gates are the landed evidence). (2) The pre-fold example's diagnostic modes
+(--denoise-only/--dump-params/--encoder-only/--save-embeds/--decode-latent/
+--roundtrip/--prompt-image/--cond-image/--dry-run, multi --ref-image) were
+deleted with the private pipeline; capabilities remain library/test-reachable;
+the GB10 speed recipe must move to the seam. (3) Server-arm numeric deltas
+disclosed above. (4) The CPU host-f32 GGUF arm is off the ABI (keep-quant is
+the gated arm). (5) /v1/videos job/status/content stay VideoJobStore-served
+(unchanged); no async-job C-ABI shape yet.

CMakeLists.txt

@@ -673,6 +673,12 @@ add_library(vllm STATIC
   # Parakeet head archs so config.json RESOLVES (SupportsTranscription mirror).
   src/vllm/multimodal/parakeet_transcription.cpp
   src/vllm/model_executor/models/parakeet_registry.cpp
+  # The ONE video-generation seam every consumer drives (C ABI vllm_video_*,
+  # the server's /v1/videos, the minimax-h3-gen example) — ARCH-ONE-SURFACE
+  # ROW 2: absorbs the assembly pipeline examples/minimax_h3_gen and the
+  # server's private /v1/videos twin. Builds artifacts + the ffmpeg argv and
+  # SPAWNS NOTHING (the ratified process boundary).
+  src/vllm/multimodal/minimax_h3_video.cpp
   src/vllm/v1/core/encoder_cache_manager.cpp
   src/vllm/tokenizer/unicode_data.cpp
   src/vllm/tokenizer/pretokenizer.cpp