feat(abi): MiniMax-H3 video+audio generation through the ONE surface — vllm_video_*, ABI v12, /v1/videos via the library seam, thin-client examples - #123
Conversation
900662c to
3e055cf
Compare
…— vllm_video_*, ABI v12, /v1/videos via the library seam, thin-client examples ONE-SURFACE fold ROW 2 (row/H3-VIDEO-ABI, task #283; the #121 precedent's shape: three-arm fold gate, guard-ratchet shrink). The library absorbs everything the examples privately owned. GOLDENS FIRST: minimax_h3_video_fold_fixture.h writes a deterministic tiny checkpoint set (857KB F32 ComfyUI-GGUF DiT at the test_minimax_h3.cpp:3786 reduced geometry, on-disk reduced ViT3D/BigVGAN VAEs + config.json, prompt embeds); the PRE-fold minimax-h3-gen binary AT THE BRANCH BASE (fc636c7) rendered it (--partition fl2va --steps 3 --frames 5 --height/width 32, CPU, keep-quant; run-to-run deterministic, diff -r clean) and its 8 frames + WAV + both minimax-h3-mux --print-only argv lines are the committed goldens in fixtures/minimax_h3_video_fold/. W1 seam: vllm::multimodal::MiniMaxH3VideoEngine absorbs the 1293-line example driver AND the server's 354-line /v1/videos twin — 4 DiT loader arms (GGUF keep-quant / dequant-bf16 host+streamed / bf16-shard stream / NVFP4 bf16+fp4-resident), VAE decoder + lazy encoder halves, the H3-Encoder tower staged once (GGUF or bf16 shards), the #77 partition guard, fl2va/ref2va conditioning, the byte-exact splitmix64 Box-Muller noise streams (VT_H3_GAUSSIAN_NOISE honored; seeded requests derive the audio stream via splitmix64), artifact writing and mux-argv assembly. The library SPAWNS NOTHING (mkdir -p became std::filesystem; ffmpeg stays caller-side — the ratified 2026-08-03 boundary). MiniMaxH3VideoGenParamsFromRequest is the ONE /v1/videos mapping so HTTP and FFI cannot drift. W2 ABI: the ratified vllm_video_* slice (one-surface-abi.md:92-108) — vllm_video_engine_load/free, vllm_video_model_params/_default, vllm_video_params/_default, vllm_video_generate, vllm_video_result/_free — plus the engine-free vllm_video_mux_argv(+_params_default,+_free) composer the mux thin client needs. VLLM_ABI_VERSION 11 -> 12; test_capi floor pin >= 12 (the == checks move with the macro). As-shipped deltas vs the proposal argued in the spec (+prompt_embeds_path/+partition/+fp4_resident/+output_dir; -task/-duration; ONE ref_image). Zero values preserve behaviour. Refusals both directions: vllm_video_engine_load on a text dir names vllm_engine_load; vllm_engine_load on the H3 dir fails byte-for-byte as captured at v11. W3 server: the VideoState block is DELETED; /v1/videos = seam Load + a runner lambda (FromRequest -> Generate -> fork/execvp(mux_argv)). Direct MiniMaxH3 refs 54 -> 7, all seam type names. DISCLOSED numeric deltas on the server arm (no goldens existed; the drift WAS the defect): legacy single-stream uniform noise -> the shared ratified recipe; host-f32 GGUF default -> keep-quant; new --video-dequant-bf16 flag. W4 examples: minimax_h3_gen (1293 -> 216 lines) + minimax_h3_mux are vllm.h+vllm::shared thin clients, BOTH byte-identical to the pre-fold binaries on the fixture (frames+WAV cmp clean; mux argv diff clean; the pre-fold diagnostic modes died with the private pipeline — named residual). Ratchet: both example-abi-allowlist rows removed, MAX_INTERNAL_REACHING 11 -> 9 with the equality pin + spec claims moved; the abi-capability video row closes; FEATURES video row -> reachable naming the four symbols. Gates: three-arm fold gate test_minimax_h3_video_fold 4 cases / 131 assertions GREEN (seam == replicated pre-fold pipeline == committed goldens, byte-identical; partition-guard + input refusals; FromRequest field-complete); test_capi 40/373 incl. the v12 golden e2e THROUGH the C marshalling; vllm_capi_c_check strict-C11 green; test_minimax_h3 75/75 unchanged; test_openai_api_server 45/566 incl. the socket-level routes-do-not-exist-without-a-runner 404 pin; test_video_api 14/14; check-surface-coverage green with the SHRINK enforced + its 46-test mutation suite. GB10 real-video re-verification through the v12 ABI is a NAMED RESIDUAL (box on the Kimi campaign); records updated (STATUS/BENCHMARKS/FEATURES/NOW/state/model-matrix + both specs). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
3e055cf to
3d501c6
Compare
…- vllm_video_*, ABI v12, library-owned /v1/videos, thin-client examples (#123) ONE-SURFACE fold ROW 2, the largest: video generation becomes reachable to embedders (LocalAI's vllm-cpp backend is the motivating consumer). The library absorbs the 1293-line example driver into a video seam; include/vllm.h gains the ratified slice - vllm_video_engine, vllm_video_model_params/_default, vllm_video_params/_default, vllm_video_generate, vllm_video_result (frame dir + WAV + mux argv: the library COMPOSES the ffmpeg command, the caller execs, src/ spawns nothing), VLLM_ABI_VERSION 11->12 floor-pinned. The server's /v1/videos routes through the same seam (VideoRunner library-owned; ~54 direct MiniMaxH3 refs dropped); minimax_h3_gen + minimax_h3_mux are rewritten as vllm.h-only thin clients; surface-guard ratchet 11->9. Correctness: three-arm fold gate on a committed deterministic fixture - library seam == replicated pre-fold pipeline == goldens captured from the pre-fold binary at the branch base - frames+WAV byte-identical through the seam, the C marshalling, and the rewritten binaries. DISCLOSED numeric change on the server arm only: pre-fold /v1/videos had drifted to single-stream UNIFORM noise + f32-host default while the ratified example used the Gaussian (torch.randn-mirror) recipe - review-confirmed by archaeology at the base SHA; the fold unifies the server onto the ratified behavior (the drift WAS the defect the ONE SURFACE directive exists to prevent). ABI-shape deltas vs the spec proposal argued in the spec's "As-shipped deltas". Two-round-capable review: fresh mutation review (seam and replica arms independently killable, ABI floor pin, zero-value contract, route gating, ratchet pin, thin-client purity, scope) returned MERGE-CLEAN with 3 minor doc findings, recorded as follow-ups. Operator re-ran fold/h3/capi/video_api/api_server on this merged tree: 5/5 SUCCESS. Residual: GB10 real-video re-verification through the v12 ABI (box running the Kimi campaign). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
|
Landed on main as c0bab82 (mudler-authored squash via commit-tree). Review: fresh mutation review MERGE-CLEAN (seam and replica arms independently killable; the server numeric-delta claim CONFIRMED by archaeology at fc636c7 — pre-fold /v1/videos used single-stream uniform noise + f32-host while the ratified example used the Gaussian recipe; the fold unifies onto ratified). Operator-run gates on the merged tree: keyed auto-merges verified byte-identical to the deterministic rebuild; the state-entry extraction was rebuilt after a first-pass slice error (caught by seam inspection, not the checker); fold/h3/capi/video_api/api_server 5/5 SUCCESS; eleven gates 00000000000 after a 40-char STATUS compaction within this PR's own row (merge composition tipped the shrink-only ratchet). Video generation is now on the one surface: ABI v12, ratchet 9, capability table 9 reachable / 2 tracked. 3 minor review findings (stale c_header_compile claim, v12 changelog placement, fixture-flag doc note) recorded as follow-ups for the next row. Residual: GB10 real-video re-verification once the box frees. |
…nner; engine==CLI 128/128, golden 122/128 profile; ABI v13 vllm_complete_tokens; thin-client example (merged ratchet 8) Branch: row/KIMI-RUNNER-FOLD (draft PR #122 — the helper claim). ARCH-ONE-SURFACE ROW 7 / kimi-linear.md §20.3→§21, task #281. The runner previously ABORTED on Kimi's KV (MambaSpec check vs config-derived zeros); the fast paged-incremental decode was CLI-only. Now Kimi decode runs THROUGH ModelRegistry::Forward on the runner's own paged state and the engine/server serve it. B1 — KV enablement: LoadHfConfig synthesizes layer_types + the GDN-group geometry from Kimi's nested linear_attn_config (configs/kimi_linear.py:34-148, 1-indexed kda_layers). ADDITIVE: explicit-field configs (qwen3_5) never enter the branch — the priority conjunct is PINNED by a both-present test (explicit fields WIN; mutation-verified RED when the conjunct is dropped); runner.cpp untouched. B2 — KDA-paged block (KdaLayerPagedBf16): vt::KdaChunkPrefill for fresh prefills (VT_KIMI_PAGED_KDA_CHUNK=0 A/B) / vt::KdaGatedDeltaRule (T==1) for decode + continuing prefills over the paged gdn_state group keyed by non_spec_state_indices (GdnStateGather/Scatter); conv taps via CausalConv1dFwd (varlen) / CausalConv1dUpdate (decode) in vLLM's conv_state.chunk(3) [q|k|v] layout. NOT per-head GdnBlockPaged — KDA's per-K-channel decay needs the KDA ops; shared GDN kernels untouched. B3 — NoPE-MLA-paged block: latent rows via vt::ConcatAndCacheMla at attn_meta.slot_mapping (bf16 pages; conv cache dtype follows ResolveKvCacheDType, mirroring kda_state_dtype). PRODUCTION arm = mla::ForwardMlaAttentionBlock (vLLM's absorbed-MQA decode / FA2 prefill, identity-RoPE, scale qk^-0.5, load-time AbsorbKvBProjBf16 into MlaResidentWeights::w_uk_t/w_uv) — VT_KIMI_PAGED_MLA_FA2 DEFAULT ON, GB10-ruled: it reproduces the golden's 122/128 near-tie profile EXACTLY. Diagnostic arm ('0') = the exact f64 island (CPU fold-identity vehicle; GB10 111/128 = the §19 GPU M-tiling near-tie class, not a paging bug). B4 — ONE SURFACE: registry loads the bf16-RESIDENT tower through the engine (ModelFactory::stage_on_load — queue before load, per-tensor stage-and-release, the §13 recipe); ForwardPaged honors device_token_ids (the DEFAULT-ON async device mirror leaves host ids stale — a measured GB10 9/128 divergence, RED-first CPU-pinned); vllm_complete_tokens (ABI v13 after the #123 video-v12 rebase; changelog slotted; floor pin >= 13; dlopen 28/28 + c_header_compile reference); examples/kimi_linear_gen REWRITTEN as a thin vllm.h + vllm::shared client; example-abi-allowlist kimi row REMOVED — merged ratchet MAX_INTERNAL_REACHING = 8 (with #123's two minimax removals), equality pin 8, coupled spec/state claims moved together; the CLI-incremental reference leg preserved as the env-gated test_kimi_linear_fold_gate. GB10 gates (dgx, /dev/shm CUDA build: CUTLASS 4.5.0 + FA2 + Triton AOT sm_121a; golden md5 bfa5bdbf; flock, drop_caches, worker parked, no reboot during the gates): - SACRED post-fold: 35B test_qwen36_paged_engine 2/2·315 PASS; 27B test_qwen27_paged_engine 1/1·235 PASS. - Gate A fold identity: CLI reference leg reproduces §19 EXACTLY (122/128 @ 18.93 tok/s, p7 got byte-equal); ENGINE leg (vllm_engine_load + vllm_complete_tokens, FA2 default) == CLI 128/128 BYTE-IDENTICAL; vs golden 122/128 — the SAME near-tie profile (>=122 bound MET). - Server smoke /v1/completions: PASS — streamed 48 tokens / 2.52 s = 19.0 tok/s wall, coherent; non-streamed coherent; tokenizer via TikTokenConverter (round-trip verified; Kimi ships tiktoken-only). - SPEED: server 19.0 tok/s wall (~0.90x the #111 vLLM ~21 floor) — >= vLLM NOT met; residual: per-step KDA host islands, grouped MoE, decode graph (spec §21). Example two-length diffs (16.9 N=64) carry cold-start pollution — measurement caveat recorded. - The in-session vLLM re-measure ABORTED BY BOX REBOOT (util 0.82 at graph capture — the §19-documented risk; not retried per the safety mandate; the #111 ~21 floor stays the denominator; box recovered clean, worker auto-restored). - CPU: test_kimi_linear_paged 8/8·206 (ForwardPaged logits BYTE-EQUAL the CLI at every step, mutation-RED on 4 seams; slot isolation; batched-prefill slot writes; device_token_ids pin; FA2-arm greedy == exact-arm + f32-page rejection; pid-unique temp fixtures); test_hf_config 18/18·187; test_capi 400; test_dlopen 28/28; full ctest 351/351 (pre-rebase base). Records: spec §21, STATUS/BENCHMARKS/FEATURES keyed rows, model-matrix row, state.md append + NOW refresh, benchmark-record append, ENVIRONMENT.md knobs (VT_KIMI_PAGED_MLA_FA2 documented default-ON, GB10-ruled). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
…odel_params.device, ABI v14, --device on server + cli ONE-SURFACE fold ROW 8 (row/DEVICE-KNOB, task #284), the smallest: an embedder can now ASK for a device instead of inheriting the accelerator-first probe. Mirrors vLLM's DeviceConfig.device names (vllm/config/device.py:13 @ 555967922; an explicit device is assigned VERBATIM and never substituted, device.py:61-66 - upstream's own --device flag was removed in v0.10, bc8a8ce5e, so the field mirrors the surviving config surface). include/vllm.h gains vllm_model_params.device (int32, 0=auto/1=cpu/ 2=cuda; 0 MUST be auto so zero-initialized structs keep the pre-v14 probe byte-identical; cpu-before-cuda follows the shipped v12 vllm_video_model_params.device precedent shifted by the auto slot). VLLM_ABI_VERSION -> 14 per the operator's version-collision heads-up (#122 reconciles the Kimi vllm_complete_tokens slice to v13 at its landing; the changelog names the gap and this branch rebases after). Floor pin advanced to >= 14 (the #121 == lesson). New vllm/config/device.h (+.cpp): vllm::Device, DeviceFromString, DeviceName. Plumb: EngineParams::device -> SelectQueue explicit arms via the pure LoadedEngine::ResolveExplicitDeviceType (explicit cpu NEVER probes; explicit cuda requires the registered kCUDA platform and fails LOUD otherwise; explicit-arm queue-creation failures propagate, unlike auto's byte-identical catch-and-fall-back). FromModelDir resolves an explicit device BEFORE any path/config I/O (DeviceConfig resolves at config time, arg_utils.py:1878), so a device error is never masked by a path error. capi validates 0/1/2 and refuses explicit cuda on the CPU-hosted transcription stack. Thin clients: --device auto|cpu|cuda on vllm-cli (ABI field only) and the server (DeviceFromString; --video-device untouched, separate engine). Also folds the #123 review's three MINOR findings (tests/docs only): c_header_compile.c actually references the v11+v12 surface + the v14 field (its "every entry point" claim went stale at v10); the v12 changelog block moved to chronological position; the H3 fold fixture flag list gained --keep-quant. Gates: full CPU build -Werror clean; test_capi (4 new v14 cases + floor), test_loaded_engine_dense (+3 device cases incl. the pure policy matrix that pins "explicit cpu beats a REGISTERED accelerator" CPU-side), test_dlopen, test_openai_api_server (+explicit-cpu serving smoke), test_minimax_h3_video_fold, test_model_loader_gguf, test_platform all green; surface-coverage green, ratchet UNTOUCHED at 9; STATUS ratchet paid by removing a stale back-to-back merge duplicate (279200 -> 279150). Ten mutations killed and reverted (defaults, zero-map, range, both plumb sites, ctor arm, unconditional fallback, policy flip, parse flip, forgotten bump); kills recorded in .agents/state.md. Residuals: CUDA-build A/B (no dgx in scope) and the pending #122 rebase, both named in the state entry. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
…odel_params.device, ABI v14, --device on server + cli ONE-SURFACE fold ROW 8 (row/DEVICE-KNOB, task #284), the smallest: an embedder can now ASK for a device instead of inheriting the accelerator-first probe. Mirrors vLLM's DeviceConfig.device names (vllm/config/device.py:13 @ 555967922; an explicit device is assigned VERBATIM and never substituted, device.py:61-66 - upstream's own --device flag was removed in v0.10, bc8a8ce5e, so the field mirrors the surviving config surface). include/vllm.h gains vllm_model_params.device (int32, 0=auto/1=cpu/ 2=cuda; 0 MUST be auto so zero-initialized structs keep the pre-v14 probe byte-identical; cpu-before-cuda follows the shipped v12 vllm_video_model_params.device precedent shifted by the auto slot). VLLM_ABI_VERSION 13 -> 14 on top of #122's vllm_complete_tokens v13 (rebased onto 9f772ca; the changelog now reads v10..v14 chronological - the v12-after-v11 move is the #123-review minor kept through the rebase). Floor pin advanced to >= 14 (the #121 == lesson). New vllm/config/device.h (+.cpp): vllm::Device, DeviceFromString, DeviceName. Plumb: EngineParams::device -> SelectQueue explicit arms via the pure LoadedEngine::ResolveExplicitDeviceType (explicit cpu NEVER probes; explicit cuda requires the registered kCUDA platform and fails LOUD otherwise; explicit-arm queue-creation failures propagate, unlike auto's byte-identical catch-and-fall-back). FromModelDir resolves an explicit device BEFORE any path/config I/O (DeviceConfig resolves at config time, arg_utils.py:1878), so a device error is never masked by a path error. capi validates 0/1/2 and refuses explicit cuda on the CPU-hosted transcription stack. Thin clients: --device auto|cpu|cuda on vllm-cli (ABI field only) and the server (DeviceFromString; --video-device untouched, separate engine). Also folds the #123 review's three MINOR findings (tests/docs only): c_header_compile.c actually references the v11+v12 surface + the v14 field (composed with #122's vllm_complete_tokens reference; its "every entry point" claim went stale at v10); the v12 changelog block moved to chronological position; the H3 fold fixture flag list gained --keep-quant. Gates: full CPU build -Werror clean; test_capi (4 new v14 cases + floor >= 14, #122's v13 cases green alongside), test_loaded_engine_dense (+3 device cases incl. the pure policy matrix that pins "explicit cpu beats a REGISTERED accelerator" CPU-side), test_dlopen, test_openai_api_server (+explicit-cpu serving smoke), test_minimax_h3_video_fold, test_model_loader_gguf, test_platform all green; surface-coverage green, allowlist ratchet UNTOUCHED at 8 (post #122); STATUS ratchet paid by removing a stale back-to-back merge duplicate (279200 -> 279150, measured 279111 post-rebase). Ten mutations killed and reverted (defaults, zero-map, range, both plumb sites, ctor arm, unconditional fallback, policy flip, parse flip, forgotten bump); kills recorded in .agents/state.md. Residual: CUDA-build A/B (no dgx in scope), named in the state entry. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
…params.device, ABI v14, --device on cli+server (#136) ONE-SURFACE fold ROW 8. vllm_model_params gains an appended device field (0=auto = today's accelerator-first probe, byte-preserving for zero-initialized callers; 1=cpu, 2=cuda), mirrored from vLLM DeviceConfig (config/device.py @ 555967922, ported as include/vllm/config/device.h + device.cpp; upstream removed the --device FLAG in v0.10 while keeping the field surface - disclosed with citation). Plumb: vllm_model_params -> EngineParams::device -> SelectQueue via pure LoadedEngine::ResolveExplicitDeviceType - explicit cpu never probes, explicit cuda on a build without CUDA fails loud with NO fallback, resolution happens before any I/O. --device auto|cpu|cuda on vllm-cli and examples/server (ABI-field-only thin clients; --video-device untouched). VLLM_ABI_VERSION 13->14, floor-pinned >= 14. Also closes the three #123 review minors: c_header_compile.c references the v11+v12+v13+v14 surface, the v12 changelog block moved chronological, the fold fixture flag list gained --keep-quant. Review: implementer 10-kill mutation battery, then a FRESH mutation review MERGE-CLEAN - zero-contract/no-fallback/never-probes/ resolve-before-IO/floor-pin all independently re-killed, the device.py port verified verbatim against the parity pin, blast radius (model_loader shared with the Kimi paged load) green. Disclosed honestly: the 0-vs-explicit-cpu distinction is CPU-tier-invisible by construction - the CUDA-build A/B is a named residual. Operator: merged tree byte-identical to the reviewed head, capi/dlopen/loaded_engine/api_server 4/4 SUCCESS. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
ONE-SURFACE fold ROW 2 (task #283) — IMPLEMENTED; single squashed commit per the doc-checkpoint per-commit obligation. DRAFT until the operator's review rounds.
What landed
minimax_h3_video_fold_fixture.hwrites a deterministic tiny checkpoint set; the PRE-foldminimax-h3-genat the branch base (fc636c7) rendered it and its 8 frames + WAV + bothminimax-h3-mux --print-onlyargv lines are committed goldens (run-to-run deterministic,diff -rclean).vllm::multimodal::MiniMaxH3VideoEngine(+MiniMaxH3VideoGenParamsFromRequest): absorbs the 1293-line example driver AND the server's 354-line /v1/videos twin. Library spawns nothing (ffmpeg boundary stands; mkdir via std::filesystem).vllm_video_engine_load/free,vllm_video_model_params(_default),vllm_video_params(_default),vllm_video_generate,vllm_video_result(_free)+vllm_video_mux_argv((_params_default)/_free); test_capi floor >= 12; refuse-both-directions pinned (video-load on a text dir names vllm_engine_load; vllm_engine_load on the H3 dir fails byte-for-byte as at v11). As-shipped deltas vs the ratified proposal argued in.agents/specs/one-surface-abi.md.vllm.h+vllm::sharedthin clients, byte-identical to the pre-fold binaries on the fixture. Allowlist −2 rows,MAX_INTERNAL_REACHING11→9 + equality pin, abi-capability video row closed, FEATURES video row → reachable.Gates (all green on 3e055cf)
test_minimax_h3_video_fold: 4 cases / 131 assertions — seam == replicated pre-fold pipeline == committed goldens, frames+WAV byte-identical; mux argv pinned.test_minimax_h375/75 unchanged.test_capi40 cases / 373 assertions incl. the v12 golden e2e THROUGH the C marshalling + zero-value contract + refusals;vllm_capi_c_checkstrict-C11 green.test_openai_api_server45/566 incl. the socket-level text-server-404s-/v1/videos pin;test_video_api14/14.scripts/agent-preflight.shEXIT=0;check-surface-coverage+ 46-test mutation suite green with the shrink enforced.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys