diff --git a/.agents/NOW.md b/.agents/NOW.md index 115ba4fe4..08aea08e8 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -24,10 +24,10 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED (A/B ties our+vLLM PTX all ptxas/JIT; +10us=engine context, not codegen)** | TERMINAL: at parity | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): async mirror on classic dense Qwen3; SACRED 184/184 | Residual: sibling scope one-liner | | CPU levers (`QUANT-GGUF-CIQ-GEMM`) | Profile DONE: decode **47% threadpool sync**, prefill **~39% paged attn**. **G5 not next** | Parakeet encoder; attn dtype hoist | -| Supported-models list (`row/DOCS-SUPPORTED-MODELS-MATRIX`) | **DRAFT PR**: FEATURES per-arch table CI-bound to registry (30 archs) | Reviewer merge | +| Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | β€” | | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning | | `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable | -| Surface coverage (`ARCH-ONE-SURFACE`) | **Guard LANDS** `check-surface-coverage.py` (2 axes): examples+FEATURES capsβ†’ABI or tracked | PR #120 | +| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 1 LANDED (#121)**: Parakeet ASR on the surface (`vllm_transcribe` v11, `/v1/audio/transcriptions`, ratchet 12β†’11) | Next fold row | In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index a34325431..4e7a56944 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -63,8 +63,8 @@ Engaged architectures (the 47 non-`INVENTORIED` rows): | βœ… | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (text path) | text-gen STRICT token-exact 315/315 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS, vision tower pending (M2/M3) so the row is `PARTIAL` (text-only) | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | | 🚧 | `Qwen3VLForConditionalGeneration` | Qwen3-VL-4B-Instruct (image + video vehicle) | **IMAGE e2e WORKING (M2 CLOSED): STRICT imageβ†’text token-exact 32/32 vs vLLM 0.25.0** (full pipeline: C++ processorβ†’M2a towerβ†’mergeβ†’forked MRoPE/DeepStack greedy decode); correctness complete, **speed pending**. Underpinned by mm processor (M1), vision tower (M2a faithful), M2b/M2c numeric contracts (85/85). **VIDEO (M3c 2026-07-25): preprocessing + full wiring LANDED + unit-gated** β€” video-processor pixel_values_videos BIT-exact 41/41 (RED-first), video MRoPE positions BIT-exact vs vLLM, per-frame windowed tower faithful (rel-L2 0.072); video e2e **NEAR-TIE-ROBUST PASS** (gate form chosen BY MEASUREMENT 2026-07-25): teacher-forcing vLLM 0.25.0 on OUR exact tokens proves the sole divergence is ONE genuine bf16 near-tie at tok22 (gap **0.125 nats**, our token vLLM's 2nd of 4 tied choices) with every downstream token == vLLM's teacher-forced argmax at gap **0.0** β€” the tower already accumulates in f32 everywhere (matches vLLM), so the residual is the irreducible bf16 envelope, NOT a fixable numeric gap. VIDEO understanding now WORKS e2e (correctness complete, **speed pending**); image e2e STRICT 32/32 unchanged (the deterministic strict-pass proof, no regression). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md Β§16): the shared vision tower now defaults to the flash-tiled `AttentionDenseFlash` (byte-identical to warp; STRICT image 32/32 HELD, nsys default 4B e2e = 24 flash-kernel inst/zero warp, goldens md5 unchanged); the tower already BEATS vLLM (~250 ms eager encode) β€” the profile REFUTED a bigger lever.** **OpenAI SERVING (2026-07-28, `CLAIM-MM-SERVING-W1`, [mm-serving.md](specs/mm-serving.md)): first CPU brick landed β€” the chat request now parses OpenAI multimodal content parts (`image_url`/`input_audio`/`audio_url`) + decodes base64/data-URI + routes to the EXISTING processor β†’ `MultiModalInputs` (gate `test_chat_mm` 5/5, CPU no-weights). This model is the named `MM-SERVE-E2E` closing gate: a real image+prompt OpenAI `/v1/chat/completions` request β†’ token-correct output vs the mm oracle (needs the engine mm plumbing `MM-SERVE-ENGINE` + DGX). Not yet wired into the OpenAI server end-to-end.** **SERVING W2 (2026-07-28, `CLAIM-MM-SERVING-W2`): `MM-SERVE-ENGINE` DONE** β€” the engine now CARRIES the parsed `MultiModalInputs`: additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate` overloads via `InputProcessor::process_inputs_mm` (mm_features onto `EngineCoreRequest`/`Request`), chat-template placeholder-STRING helpers, serving_chat `MultiModalChatFn` seam (default unset β‡’ text byte-identical); CPU gate `test_input_processor` +2 + `test_chat_mm` +2 (full chain parseβ†’routeβ†’engine request with the 196-slot expanded prompt + mm handles). **SERVING W3 (2026-07-28, `CLAIM-MM-SERVING-E2E`): the `MM-SERVE-E2E` CPU SEAM BODY DONE** β€” `MakeQwen3VLImageChatFn` (chat_mm.{h,cpp}) turns an image chat request into the placeholder-EXPANDED engine input (marker-inject β†’ chat template β†’ `EncodeWithSpecialTokens` single image_pad markerβ†’one image_token_id β†’ `RouteImageRgb` EXPAND to 196 image tokens + mm_features), wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text path byte-identical). Gate `test_chat_mm` 8/8 (RED text path = 0 image tokens) + `test_openai_serving` (production seam invoked + routed). **The GPU e2e forward is ARCHITECTURALLY BLOCKED (not box contention):** the engine model runner has no mm forward β€” `ModelForwardInput` has no vision field, `runner.cpp` ignores `Request.mm_features`, this arch is UNREGISTERED (`REGISTER_VLLM_MODEL` absent), and the M2c `Qwen3VLGenerateGreedy` is a standalone driver. Fold-into-engine recipe (add a vision-embed field β†’ runner runs the tower via the encoder cache + merge β†’ register the arch β†’ dgx token-exact vs the M2c golden) in `specs/mm-serving.md`. **ENGINE MM-FORWARD LANDED (2026-07-28, `CLAIM-ENGINE-MM-FORWARD`): the block is RESOLVED β€” `ModelForwardInput` gains an additive default-nullopt `mm` field (merged inputs_embeds + 3-D MRoPE + DeepStack), Qwen3-VL is now `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`, `supports_multimodal=true`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through `ModelRegistry::Forward`). Text inertness (RED line, shared-path change): `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green (nullopt-for-text β‡’ byte-identical). Engine mm-forward token-exact gate `test_qwen3vl_registry_e2e` (imageβ†’text THROUGH `ModelRegistry::Forward` == M2c golden STRICT, dgx.casa GB10). Residual: the in-runner scheduler-fed tower run + the full server `/v1/chat/completions` GPU e2e.** | `MODEL-MM-qwen3-vl-qwen3-vlfor-conditional-generation` | | 🚧 | `VoxtralForConditionalGeneration` | Voxtral-Mini-3B-2507 (AUDIOβ†’text, audio-track A3) | **FIRST e2e AUDIO UNDERSTANDING: audioβ†’text vs vLLM 0.25.0** β€” A2 Whisper-large-v3 encoder at Voxtral config (128 mel/1280/32L/head_dim 64) + AudioLanguageAdapter projector + masked-scatter merge into the LANDED Mistral/Llama decoder (untied lm_head, mistral-consolidated q/k rope-permute) β†’ forked greedy. Gate form BY MEASUREMENT: vLLM greedy K=5 deterministic β‡’ STRICT is the bar; STRICT prefix 33/48 exact vs vLLM greedy, then the ratified near-tie-robust gate PASSES (worst teacher-forced gap 0.0 nats, sole branch = a 4-way bf16 tie at -2.069 nats) β€” bit-exact infeasible (encoder uses different bf16 GEMM/attn kernels). Decoder proven token-exact (ref-audioβ†’48/48). Additive driver/loader gated on audio β‡’ Mistral text byte-identical. **Speed pending** (decode now graph-captured via `VoxtralDecodeGraph`, lever #3 W1 2026-07-27: small real win 60.94 vs 61.71 ms/tok, still ~1.49Γ— vs vLLM 40.8 ms). **Decode-kernel efficiency ATTRIBUTED + VALIDATED ceiling (2026-07-27, `CLAIM-MM-SPEED-DECODE-KERN`, multimodal-speed.md Β§11):** the whole ~20 ms/tok residual is the naive scalar `PagedAttentionKernel` decode attention (723 Β΅s Γ— 30 layers = 21.7 ms/step, ~120Γ— the KV floor); the 1:1 vLLM lever (FA2 `flash_attn_varlen` decode) is already in-binary, gated off only because the driver's single KV block (444) isn't Γ·16. `block_sizeΓ·16` β†’ decode via FA2: TPOT **59.4β†’38.2 ms/tok (βˆ’21.2, ~36%) = 0.94Γ— vLLM 40.8 ms β€” BEATS parity**, and the FA2 sequence is a VALID vLLM greedy branch (teacher-force PASS, 0 divergences, gap 0.0). But it flips the committed near-tie golden's exact-tie branch (`repro` 48β†’18) β†’ blocked byte-exact (RED line). bf16 near-tie / golden-pinning ceiling; RECORDS-ONLY (14/14 held, golden unchanged); win reachable via `block_sizeΓ·16` + golden regen. **ADOPTED 2026-07-27 (USER-APPROVED, `CLAIM-MM-SPEED-DECODE-KERN-ADOPT`, multimodal-speed.md Β§12): FA2 decode SHIPS as the Voxtral default β€” audio DECODE now BEATS vLLM (0.97Γ—), the LAST mm decode-speed gap CLOSED.** One-line `block_sizeΓ·16` routes decode via FA2 `LaunchDecodeVarlenFA2Bf16` (nsys: `flash_fwd_splitkv` 1410 @ 18.5 Β΅s, ZERO `PagedAttentionKernel`). `test_voxtral_e2e` converted to the ratified near-tie DISTRIBUTIONAL gate (binding = teacher-force PASS, KERNEL-INDEPENDENT; strict prefix exact to the first bf16 tie β€” FA2 takes the other side of the pos-18 2-way EXACT tie β‡’ prefix 18, `>=18`; determinism anchor to the FA2 seq). `voxtral_neartie.json` md5 `3d199c2d…`β†’`937b9ad3…`; STRICT golden `voxtral_golden.json` `8ab87b7e…` UNCHANGED. **Gate PASS 16/16**; teacher-force vLLM 0.25.0 on the FA2 seq = **0 divergent, worst gap 0.0000 nats, PASS**. Capture-safe (graph S=1 + 46 replays valid; compute-sanitizer **0 errors** on the graphed-FA2-decode path; 3 runs byte-identical) β‡’ ships as the DEFAULT graph path. **A/B (same-binary, 6 reps rep0 dropped, steady-state):** scalar 60.50 β†’ FA2 **39.50 ms/tok** (βˆ’21.0, ~35%, NON-OVERLAPPING) = **0.97Γ— vLLM 40.8 ms β€” BEATS**. Audio DECODE now correctness- AND speed-DONE; row stays `PARTIAL`/`ACTIVE` β€” umbrella DONE bar has audio TTFT (32-layer Whisper encoder, UNMEASURED vs vLLM 43 ms) + c2+ batched / `audio_url` serving ingestion still open (same as image/video). **ENCODER TTFT MEASURED + warp-attention brick 2026-07-27 (`CLAIM-MM-SPEED-AUDIO-ENC`, multimodal-speed.md Β§13):** the Whisper encoder ran the naive `kAttention` (O(tΒ²) per-key block-`__syncthreads`) β€” routed the encoder self-attention (hd-64, non-causal) to the warp-scoped `vt::AttentionDenseFast` (Β§7's vision-tower fix; `kAttention` untouched β‡’ text byte-identical). Encoder forward **8870β†’1890 ms (4.7Γ—, NON-OVERLAPPING)**; `test_voxtral_e2e` **16/16** with ZERO token flips (naive arm also 16/16 same tokens; goldens md5 UNCHANGED), proof-of-run nsys `AttentionWarpKernel` 32 inst / zero naive. **NOT at parity:** ~1.89 s vs vLLM's 43 ms (~44Γ—) β€” the warp kernel is STILL 31.8 ms/layer (O(tΒ²), memory-bound on redundant K/V reads); closing needs a flash-TILED non-causal hd-64 attention (LARGE) + resident one-time encoder weights (MEDIUM, byte-exact). **ENCODER FLASH-TILED KERNEL LANDED 2026-07-28 (`CLAIM-MM-SPEED-AUDIO-ENC-KERNEL`, multimodal-speed.md Β§14):** Β§13 lever #1 done β€” new `vt::AttentionDenseFlash` (`OpId::kAttentionDenseFlash`, `cuda_ops.cu`) shared-memory-TILES K/V across a block of `kFlashBr=16` query-warps (FA2 K/V tiling, structure-ported from vendored `flash_fwd_kernel.h:52`), per-warp math copied verbatim from `AttentionWarpKernel` β‡’ BIT-IDENTICAL; encoder routed to it by default (`VT_WHISPER_ENC_WARP=1`/`EAGER=1` A/B); `kAttention`/`kAttentionDenseFast` untouched β‡’ text/vision byte-identical. **16/16** default-flash, flash/warp/eager token md5-IDENTICAL (`89923566…`) β‡’ ZERO flips, goldens md5 UNCHANGED, nsys proof `AttentionDenseFlashKernel` 32 inst / zero warp+naive, RED confirmed, compute-sanitizer 0, 3 runs identical. **A/B same-binary:** attention **35.11β†’19.29 ms/layer (1.82Γ—)**, encoder forward **~1834β†’~1375 ms (1.33Γ—)**. **NOT at parity:** ~1.37 s vs vLLM ~43 ms (~32Γ—, was ~44Γ—) β€” the scalar warp-per-query recurrence is now serial-latency-bound (L2 already served the redundant reads β‡’ 1.8Γ— not 16Γ—); gap-closer = tensor-core MMA hd-64 non-causal FA2 (LARGE) + resident encoder weights (MEDIUM). **ENC lever #2 LANDED 2026-07-28 (`CLAIM-MM-SPEED-AUDIO-ENC-RESIDENT`, multimodal-speed.md Β§15): device-resident one-time encoder weights** (mirror the Qwen `d_dev` residency seam; each of the 487 encoder weight tensors f32β†’bf16-converted + H2D-uploaded ONCE into a `mutable shared_ptr` handle, reused across forwards). BYTE-EXACT (moves data, not math): `test_voxtral_e2e` **16/16**, goldens md5 UNCHANGED (`8ab87b7e…`/`937b9ad3…`), compute-sanitizer 0 errors. **Proof-of-run/RED (`VT_WHISPER_ENC_REMARSHAL`):** nsys HtoD resident 740 ops/9.4 GB vs re-marshal 1714 ops/11.9 GB (βˆ’974 ops, βˆ’2.5 GB). **A/B same-binary (`flock`, 6 reps rep0 dropped):** encoder forward **~1377β†’~729 ms (βˆ’648 ms, 1.89Γ—)** β€” removes the confirmed ~0.75 s per-call host marshalling. Trajectory 1834β†’1375β†’729 ms; vs vLLM ~43 ms now ~17Γ— (was ~32Γ—). NOT at parity β€” residual 729 ms is GPU-compute-bound (scalar warp attention); LARGE gap-closer remains tensor-core MMA hd-64 FA2. Audio TTFT/encoder stays speed-pending | `MODEL-MM-voxtral-voxtral-for-conditional-generation` | -| 🚧 | `ParakeetForCTC` (encoder + CTC head) | Parakeet / FastConformer ASR: an audio COMPONENT, not a registry arch (vLLM wraps transformers' `ParakeetEncoder` inside `nano_nemotron_vl.py`) | P4 LANDED 2026-08-07 on CPU: encoder + CTC greedy + log-mel front end, gated stage-by-stage against a dumped HuggingFace `ParakeetForCTC` oracle (rel-L2 <= 3.5e-7) with greedy and collapsed token ids EXACT, plus an independent in-test reference and 20/20 mutants killed. Real transcript verified 2026-08-07 on `nvidia/parakeet-ctc-0.6b` AND `-ctc-1.1b`; no CUDA provider, no GPU suite, no aarch64, no speed number | `MODEL-AUDIO-PARAKEET-ENCODER` | -| 🚧 | `ParakeetForRNNT` / `ParakeetForTDT` (transducer heads) | Parakeet RNN-T and TDT ASR: the decode half of the same family, likewise an audio COMPONENT and not a registry arch (vLLM has no transducer call site at all) | P6 LANDED 2026-08-07 on CPU: LSTM prediction network, joint network, TDT duration head and the greedy transducer loop, gated against dumped HuggingFace `ParakeetForRNNT` / `ParakeetForTDT` oracles from transformers `main` with the emitted sequence and per-step durations EXACT, plus an independent in-test LSTM reference. Real transcripts verified on `nvidia/parakeet-rnnt-0.6b` and `-tdt-0.6b-v3`, token ids EXACT vs HF `generate()` end to end. **Corrects the P4 record**, which called the transducer unmirrored off the locally installed transformers 5.3.0. No CUDA, no GPU suite, no aarch64, no speed number; `.nemo`-only checkpoints out of reach | `MODEL-AUDIO-PARAKEET-TRANSDUCER` | +| 🚧 | `ParakeetForCTC` (encoder + CTC head) | Parakeet / FastConformer ASR: an audio COMPONENT, not a registry arch (vLLM wraps transformers' `ParakeetEncoder` inside `nano_nemotron_vl.py`) | P4 LANDED 2026-08-07 on CPU: encoder + CTC greedy + log-mel front end, gated stage-by-stage against a dumped HuggingFace `ParakeetForCTC` oracle (rel-L2 <= 3.5e-7) with greedy and collapsed token ids EXACT, plus an independent in-test reference and 20/20 mutants killed. Real transcript verified 2026-08-07 on `nvidia/parakeet-ctc-0.6b` AND `-ctc-1.1b`; no CUDA provider, no GPU suite, no aarch64, no speed number. **ONE-SURFACE ROW 1 (2026-08-07): NOW a registered arch** β€” `ParakeetForCTC` self-registers (BEYOND-PIN: pinned vLLM has Parakeet only as the NemotronH audio component, registry.py:511-513) with the SupportsTranscription-only ModelInfo; text paths refuse by task; served via `vllm_transcribe` (ABI v11) + `/v1/audio/transcriptions`; example = thin `vllm.h` client, transcript byte-identical to pre-fold | `MODEL-AUDIO-PARAKEET-ENCODER` | +| 🚧 | `ParakeetForRNNT` / `ParakeetForTDT` (transducer heads) | Parakeet RNN-T and TDT ASR: the decode half of the same family, likewise an audio COMPONENT and not a registry arch (vLLM has no transducer call site at all) | P6 LANDED 2026-08-07 on CPU: LSTM prediction network, joint network, TDT duration head and the greedy transducer loop, gated against dumped HuggingFace `ParakeetForRNNT` / `ParakeetForTDT` oracles from transformers `main` with the emitted sequence and per-step durations EXACT, plus an independent in-test LSTM reference. Real transcripts verified on `nvidia/parakeet-rnnt-0.6b` and `-tdt-0.6b-v3`, token ids EXACT vs HF `generate()` end to end. **Corrects the P4 record**, which called the transducer unmirrored off the locally installed transformers 5.3.0. No CUDA, no GPU suite, no aarch64, no speed number; `.nemo`-only checkpoints out of reach. **ONE-SURFACE ROW 1 (2026-08-07): `ParakeetForRNNT`/`ParakeetForTDT` registered** (BEYOND-PIN, transcription-only refuse-by-task) and served through the same `vllm_transcribe` / `/v1/audio/transcriptions` / thin-client surface as the CTC head | `MODEL-AUDIO-PARAKEET-TRANSDUCER` | | βœ… | `OPTForCausalLM` | OPT-125m | STRICT token-exact 6/6 vs vLLM 0.25.0; speed pending | `MODEL-TEXT-opt-optfor-causal-lm` | | βœ… | `DeepseekV2ForCausalLM` | DeepSeek-V2-Lite (MLA) | SACRED gate 8/8 token-exact vs vLLM 0.25.0; speed short (attributed, W9) | `MODEL-TEXT-deepseek-v2-deepseek-v2-for-causal-lm` | | βœ… | `LlamaForCausalLM` | Llama-3.2-1B dense (+ Yi + `InternLM3ForCausalLM` aliases) | STRICT token-exact 16/16 vs vLLM 0.25.0; speed pending. Llama-alias checkpoints gated 2026-07-26: Yi (`01-ai/Yi-Coder-1.5B-Chat`, arch=LlamaForCausalLM, zero delta) 16/16; InternLM3 (`internlm3-8b-instruct`, one alias line, plain-Llama+dynamic rope) 16/16 β€” CLOSES the recent-dense TEXT tier | `MODEL-TEXT-llama-llama-for-causal-lm` | @@ -476,7 +476,7 @@ of their own and are NOT part of the 328 architecture rows counted above. | ID | Item / architecture aliases | Upstream | Task / modality | Dependencies | Spike/spec | State | Our code + tests/evidence | Owner | |---|---|---|---|---|---|---|---|---| -| `MODEL-AUDIO-PARAKEET-ENCODER` | `ParakeetEncoder` + `ParakeetForCTC` (Parakeet / FastConformer conformer audio encoder + CTC head) | **NOT a registry architecture, and NOT vLLM-implemented:** `vllm/model_executor/models/parakeet.py:14,61` imports and instantiates `transformers.ParakeetEncoder` as the audio component of `nano_nemotron_vl.py` (`registry.py:511-513`). The honest mirror source is therefore transformers 5.3.0 `transformers/models/parakeet/modeling_parakeet.py` (`ParakeetEncoderRelPositionalEncoding:51`, `ParakeetEncoderFeedForward:101`, `ParakeetEncoderConvolutionModule:116`, `ParakeetEncoderAttention:259`, `ParakeetEncoderSubsamplingConv2D:357`, `ParakeetEncoderBlock:426`, `ParakeetEncoder:549`, `ParakeetForCTC:675`, plus `tokenization_parakeet.py ParakeetTokenizer._decode:28-49`); the vLLM-NATIVE halves are `ParakeetProjection:27`, `ProjectedParakeet:48` and `ParakeetExtractor:138`, with `vllm/transformers_utils/configs/parakeet.py` `ParakeetConfig:8` / `ExtractorConfig:41` | ASR / audio -> text (CTC greedy) | `vt::Conv2d`, `vt::DepthwiseConv1d`, `vt::AttentionRelPos` (spike P1-P3, LANDED `1ace9e27`) | [parakeet-conformer-encoder](specs/parakeet-conformer-encoder.md) work item P4 | `ACTIVE` | **Encoder + CTC head + log-mel front end LANDED (CPU, f32) 2026-08-07.** Code: `include/vllm/model_executor/models/parakeet_encoder.h` + `src/vllm/model_executor/models/parakeet_encoder.cpp` (subsampling -> rel-pos table -> N x block{FF/2, rel-pos attention, convolution module, FF/2, norm} -> CTC head -> greedy collapse), `src/vllm/model_executor/models/parakeet_weights.cpp` (HF-format safetensors loader), `include/vllm/multimodal/parakeet_audio_processor.h` + `src/vllm/multimodal/parakeet_audio_processor.cpp` (preemphasis -> STFT -> slaney mel -> log -> per-bin normalisation). Gates, all CPU x86-64 `-DVLLM_CPP_CUDA=OFF`: `tests/vllm/models/test_parakeet_encoder.cpp` 7 cases / 543 assertions vs an INDEPENDENT in-test double-precision reference that performs `_rel_shift` LITERALLY (encoder rel-L2 1.9e-7 to 2.4e-7, CTC logits 2.0e-7, greedy + collapsed ids EXACT); `tests/vllm/models/test_parakeet_ctc_engine.cpp` 2 / 12485 vs a dumped HuggingFace `ParakeetForCTC` oracle stage by stage (subsampling / ff1 / attn / conv / block0 / block1 / hidden / logits all rel-L2 <= 3.5e-7) with greedy and collapsed token ids EXACT on a full-length AND a PADDED batch item; `tests/vllm/multimodal/test_parakeet_audio_processor.cpp` 6 / 41054 (mel bank BIT-IDENTICAL to transformers `mel_filter_bank`, log-mel rel-L2 4.9e-6 vs `ParakeetFeatureExtractor`). 20 single-line mutations compiled and run, 20 killed, 0 survivors. **NOT verified:** no pretrained `nvidia/parakeet-*` checkpoint was downloaded, so NO transcript is claimed (the oracle is a seeded, randomly-initialised `ParakeetForCTC`; the pretrained arm is wired behind `VLLM_PARAKEET_CKPT` and SKIPS when unset); no CUDA provider exists for the three kernels, no GPU suite was run, no aarch64 run, no speed or memory number measured, claimed or owed. **Pretrained transcript ADDED 2026-08-07** on `nvidia/parakeet-ctc-0.6b` and `nvidia/parakeet-ctc-1.1b` (the 1.1b is the identical architecture at 42 layers instead of 24; downloaded, transcribed, then deleted to give the shared disk back). **The "RNN-T / TDT has no upstream in either vLLM or HF" line this row used to carry was WRONG**: it measured the locally installed transformers 5.3.0, not `main`; see `MODEL-AUDIO-PARAKEET-TRANSDUCER` below | Claude Code (opus-5), `CLAIM-PARAKEET-MODEL-P4` | +| `MODEL-AUDIO-PARAKEET-ENCODER` | `ParakeetEncoder` + `ParakeetForCTC` (Parakeet / FastConformer conformer audio encoder + CTC head) | **NOT a registry architecture, and NOT vLLM-implemented:** `vllm/model_executor/models/parakeet.py:14,61` imports and instantiates `transformers.ParakeetEncoder` as the audio component of `nano_nemotron_vl.py` (`registry.py:511-513`). The honest mirror source is therefore transformers 5.3.0 `transformers/models/parakeet/modeling_parakeet.py` (`ParakeetEncoderRelPositionalEncoding:51`, `ParakeetEncoderFeedForward:101`, `ParakeetEncoderConvolutionModule:116`, `ParakeetEncoderAttention:259`, `ParakeetEncoderSubsamplingConv2D:357`, `ParakeetEncoderBlock:426`, `ParakeetEncoder:549`, `ParakeetForCTC:675`, plus `tokenization_parakeet.py ParakeetTokenizer._decode:28-49`); the vLLM-NATIVE halves are `ParakeetProjection:27`, `ProjectedParakeet:48` and `ParakeetExtractor:138`, with `vllm/transformers_utils/configs/parakeet.py` `ParakeetConfig:8` / `ExtractorConfig:41` | ASR / audio -> text (CTC greedy) | `vt::Conv2d`, `vt::DepthwiseConv1d`, `vt::AttentionRelPos` (spike P1-P3, LANDED `1ace9e27`) | [parakeet-conformer-encoder](specs/parakeet-conformer-encoder.md) work item P4 | `ACTIVE` | **Encoder + CTC head + log-mel front end LANDED (CPU, f32) 2026-08-07.** Code: `include/vllm/model_executor/models/parakeet_encoder.h` + `src/vllm/model_executor/models/parakeet_encoder.cpp` (subsampling -> rel-pos table -> N x block{FF/2, rel-pos attention, convolution module, FF/2, norm} -> CTC head -> greedy collapse), `src/vllm/model_executor/models/parakeet_weights.cpp` (HF-format safetensors loader), `include/vllm/multimodal/parakeet_audio_processor.h` + `src/vllm/multimodal/parakeet_audio_processor.cpp` (preemphasis -> STFT -> slaney mel -> log -> per-bin normalisation). Gates, all CPU x86-64 `-DVLLM_CPP_CUDA=OFF`: `tests/vllm/models/test_parakeet_encoder.cpp` 7 cases / 543 assertions vs an INDEPENDENT in-test double-precision reference that performs `_rel_shift` LITERALLY (encoder rel-L2 1.9e-7 to 2.4e-7, CTC logits 2.0e-7, greedy + collapsed ids EXACT); `tests/vllm/models/test_parakeet_ctc_engine.cpp` 2 / 12485 vs a dumped HuggingFace `ParakeetForCTC` oracle stage by stage (subsampling / ff1 / attn / conv / block0 / block1 / hidden / logits all rel-L2 <= 3.5e-7) with greedy and collapsed token ids EXACT on a full-length AND a PADDED batch item; `tests/vllm/multimodal/test_parakeet_audio_processor.cpp` 6 / 41054 (mel bank BIT-IDENTICAL to transformers `mel_filter_bank`, log-mel rel-L2 4.9e-6 vs `ParakeetFeatureExtractor`). 20 single-line mutations compiled and run, 20 killed, 0 survivors. **NOT verified:** no pretrained `nvidia/parakeet-*` checkpoint was downloaded, so NO transcript is claimed (the oracle is a seeded, randomly-initialised `ParakeetForCTC`; the pretrained arm is wired behind `VLLM_PARAKEET_CKPT` and SKIPS when unset); no CUDA provider exists for the three kernels, no GPU suite was run, no aarch64 run, no speed or memory number measured, claimed or owed. **Pretrained transcript ADDED 2026-08-07** on `nvidia/parakeet-ctc-0.6b` and `nvidia/parakeet-ctc-1.1b` (the 1.1b is the identical architecture at 42 layers instead of 24; downloaded, transcribed, then deleted to give the shared disk back). **The "RNN-T / TDT has no upstream in either vLLM or HF" line this row used to carry was WRONG**: it measured the locally installed transformers 5.3.0, not `main`; see `MODEL-AUDIO-PARAKEET-TRANSDUCER` below. **ONE-SURFACE ROW 1 LANDED 2026-08-07** (`row/PARAKEET-ONE-SURFACE`): the "NOT a registry architecture" clause above is SUPERSEDED β€” `parakeet_registry.cpp` registers `ParakeetForCTC` (+RNNT/TDT) as SupportsTranscription-ONLY (interfaces.py:1110-1118 mirror; every text hook refuses actionably), the pipeline moved into `vllm::multimodal::ParakeetTranscriber`, `vllm_transcribe` landed on ABI v11, `/v1/audio/transcriptions` serves task-conditionally, the example is a thin `vllm.h` client, and the fold gate holds ids+text byte-identical to the pre-fold binary on committed synthetic fixtures (`tests/vllm/models/fixtures/parakeet_e2e`, goldens captured from main@f98e1e48) | Claude Code (opus-5), `CLAIM-PARAKEET-MODEL-P4`; ROW 1 fold: Claude Code (fable-5), PR #121 | | `MODEL-AUDIO-PARAKEET-TRANSDUCER` | `ParakeetForRNNT` + `ParakeetForTDT` (Parakeet RNN-T / TDT transducer heads over the same FastConformer encoder) | **NOT a registry architecture, and NOT vLLM-implemented: vLLM has no transducer call site at all** (it wraps only the ENCODER, as the audio tower of `nano_nemotron_vl.py`). The mirror source is transformers `main` `transformers/models/parakeet/modeling_parakeet.py` (`ParakeetRNNTDecoder:831`, `ParakeetRNNTJointNetwork:879`, `ParakeetForRNNT:922`, `ParakeetTDTJointNetwork:1035`, `ParakeetForTDT:1052`) plus `generation_parakeet.py` (`ParakeetRNNTDecoderCache:23`, `ParakeetRNNTGenerationMixin:125`, `ParakeetTDTGenerationMixin:271`) and `configuration_parakeet.py` (`ParakeetRNNTConfig:136`, `ParakeetTDTConfig:188`) | ASR / audio -> text (greedy transducer decode, with TDT frame skipping) | `MODEL-AUDIO-PARAKEET-ENCODER` (composes over its forward), `vt::MatmulBT` | [parakeet-conformer-encoder](specs/parakeet-conformer-encoder.md) work item P6 | `ACTIVE` | **LANDED (CPU, f32) 2026-08-07.** Code: `include/vllm/model_executor/models/parakeet_transducer.h` + `src/vllm/model_executor/models/parakeet_transducer.cpp` (embedding -> stacked LSTM prediction network with the cache's blank fast path -> decoder projector; joint `head(relu(encoder_projected + decoder))`; the greedy loop with the RNN-T `max_symbols_per_step` forced advance and the TDT duration-driven frame skip), plus the transducer half of `src/vllm/model_executor/models/parakeet_weights.cpp`. Gates, all CPU x86-64 `-DVLLM_CPP_CUDA=OFF`: `tests/vllm/models/test_parakeet_transducer.cpp` 3 cases / 777 assertions: the LSTM cell vs an INDEPENDENT in-test scalar reference over 5 shapes x bias/no-bias x 6 sequential steps; decoder and joint tensors rel-L2 <= 1.2e-7 vs a dumped HF oracle over a fixed token walk that drives the blank fast path; **the emitted sequence and the per-step durations EXACT** for BOTH `ParakeetForRNNT` and `ParakeetForTDT`, from the oracle's own projected encoder output AND end to end from mel features. The fixture asserts its own branch coverage (blank emissions, non-blank emissions, tokens held at one frame, RNN-T forced advances), so a regeneration that degenerated to one branch fails. **PRETRAINED, and this is the binding claim:** on `nvidia/parakeet-rnnt-0.6b` and `nvidia/parakeet-tdt-0.6b-v3` our emitted token ids are EXACT against a real HF `generate()` run on a LibriSpeech clip, both through the reference's own `input_features` and end to end through our own log-mel front end. **TRACED, not read (T0):** a forward hook over a full `generate()` run records that every decoder call took input_ids of shape [1,1]; the gate asserts it, because the whole greedy loop rests on it. **NOT verified:** any GPU path, aarch64, and any speed or memory number. **Checkpoint reach:** the HF-safetensors models are in; `.nemo`-only ones need upstream's `convert_nemo_to_hf.py`, and the hybrid TDT-CTC and streaming EOU models stay OUT (spike Β§ Checkpoint reach) | Claude Code (opus-5), `CLAIM-PARAKEET-MODEL-P4` | ## MODEL-SPEC - Speculative decoding models diff --git a/.agents/specs/one-surface-abi.md b/.agents/specs/one-surface-abi.md index 9c41a4f8a..4344c7111 100644 --- a/.agents/specs/one-surface-abi.md +++ b/.agents/specs/one-surface-abi.md @@ -1,6 +1,6 @@ # ONE SURFACE β€” every capability ships through the C ABI -Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE, remediation NOT started.** +Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS β€” ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11.** ## The defect @@ -34,9 +34,12 @@ stay OPEN. The complete, code-grounded matrix (all 30 archs + every off-registry | DeepSeek-V4 | YES | KV spec is a "never exercised" stub; registry forward discards attn_meta/kv | `examples/deepseek_v4_gen` | 240 lines | This table also UNDER-COUNTS: it omits Parakeet ASR (a 5th off-surface capability, landed -`fd2259d8`) and the partial internal-reachers `minimax_h3_mux`, `bench`, `tokenize`, -`dump_container`, `dequant_nvfp4`, `quant_gemm_bench` β€” 12 of 13 example binaries include -non-public headers today (only `examples/cli` is clean). The full list is the audit spec. +`fd2259d8` β€” **CLOSED 2026-08-07 by ROW 1**: `vllm_transcribe` on ABI v11, live +`/v1/audio/transcriptions`, registry refuse-by-task, example rewritten as a `vllm.h` +client, ratchet 12 -> 11) and the partial internal-reachers `minimax_h3_mux`, `bench`, +`tokenize`, `dump_container`, `dequant_nvfp4`, `quant_gemm_bench` β€” 11 of 13 example +binaries still include non-public headers (`examples/cli` and `examples/parakeet_transcribe` +are the clean ABI clients). The full list is the audit spec. `examples/server/main.cpp` also carries ~32 internal includes (54 direct `MiniMaxH3` references), i.e. it re-implements wiring rather than consuming a library entry point. So @@ -137,5 +140,11 @@ embedder produce an MP4 without reinventing the command: the library composes it ## NOT claimed -No ABI work has been done. The shape above is a proposal, not an implementation, -and it has not been reviewed against an embedder other than LocalAI. +The VIDEO ABI shape above is a proposal, not an implementation, and it has not +been reviewed against an embedder other than LocalAI. The ONE landed slice is +audio transcription (ROW 1, 2026-08-07): `vllm_transcribe` + params/result +structs on ABI v11, the `ParakeetTranscriber` library seam, task-conditional +`/v1/audio/transcriptions`, registry refuse-by-task, and the example as a thin +`vllm.h` client β€” gated byte-identical to the pre-fold transcripts. Video, +Laguna/DeepSeek/Kimi fast decode, embeddings and multimodal input remain open +rows of this program. diff --git a/.agents/specs/surface-coverage-2026-08-07.md b/.agents/specs/surface-coverage-2026-08-07.md index 94737245a..cde95df33 100644 --- a/.agents/specs/surface-coverage-2026-08-07.md +++ b/.agents/specs/surface-coverage-2026-08-07.md @@ -27,7 +27,7 @@ The four surfaces, and the public boundary the guard draws: | 1 | **MiniMax-H3 video+audio generation** | NO (off-registry diffusion) | `/v1/videos` ONLY via an example-injected `VideoRunner` | NO | `examples/minimax_h3_gen` + `examples/minimax_h3_mux` (whole pipeline) | | 2 | **Laguna fast decode** | yes, but forward is a stub | no (stub `VT_CHECK`s non-bf16) | no | `examples/laguna_gen` (keep-quant GGUF + NVFP4 W4A4) | | 3 | **DeepSeek-V4 fast decode** | yes, but forward is a W3 stub | no (stub) | no | `examples/deepseek_v4_gen` (keep-quant GGUF) | -| 4 | **Audio transcription** | NO (Whisper/Voxtral/Parakeet off-registry) | no (`/v1/audio/*` = batch residual) | no | `examples/parakeet_transcribe`; Whisper/Voxtral test-only | +| 4 | **Audio transcription** | **CLOSED (ROW 1)**: Parakeet CTC/RNNT/TDT registered (transcription-only; Whisper/Voxtral still off-registry) | **live `/v1/audio/transcriptions`** (task-conditional; the run_batch line stays a residual) | **`vllm_transcribe` (ABI v11)** | library seam `ParakeetTranscriber`; example is a clean ABI client | | 5 | **Kimi-Linear incremental decode** | yes (recompute forward IS shared) | recompute only | no | `examples/kimi_linear_gen` (Β§18/Β§19 paged-incremental + resident loader) | | 6 | **Embeddings / pooling** | NO (all `is_pooling_model=false`) | no (`/v1/embeddings` = residual) | no | engine-side pooler exists (`ENG-POOLER-SEQ`), never invoked live | | 7 | **Multimodal input over HTTP/ABI** | 5 archs `supports_multimodal` | image seam only; tower not run in engine step | no (text-only chat) | `chat_mm.cpp` seam; towers test-only | @@ -61,7 +61,7 @@ All three drivers run a PRIVATE host-argmax greedy loop, not the on-GPU sampler. | Lane | Registered | Code (file:line) | Server | C-ABI | Driver | |---|---|---|---|---|---| | MiniMax-H3 video+audio GEN | NO | `minimax_h3*.cpp` (~22 TUs), vt op `kMiniMaxH3` | `/v1/videos` via `set_video_runner` (example-injected only; `api_server.h:167`, `api_server.cpp:818`; library never sets it) | NO | `examples/minimax_h3_gen`, `examples/minimax_h3_mux` | -| Parakeet/FastConformer ASR | NO (grep `parakeet_*.cpp` for `REGISTER_VLLM_MODEL` = 0) | `parakeet_encoder.cpp`, `parakeet_transducer.cpp`, `parakeet_audio_processor.cpp`; owns transcript logic (`ReadWav16BitMono`, `LoadVocab`, `DecodeIds` Metaspace decoder β€” bypasses `vllm::Tokenizer`) | NO | NO | `examples/parakeet_transcribe` (#89) | +| Parakeet/FastConformer ASR | **YES (ROW 1)**: ParakeetForCTC/RNNT/TDT, `parakeet_registry.cpp` (SupportsTranscription-only; text paths refuse by task) | `parakeet_transcription.cpp` seam composes encoder/transducer/audio-processor; the example's private `ReadWav16BitMono`/`LoadVocab`/`DecodeIds` are DELETED (`vllm::Tokenizer` now decodes Metaspace split=true) | **`/v1/audio/transcriptions`** (task-conditional) | **`vllm_transcribe` (ABI v11)** | `examples/parakeet_transcribe` = thin `vllm.h` client | | Voxtral audio->text | NO (`VoxtralForConditionalGeneration` unregistered) | `voxtral.cpp` (`vllm::multimodal`) | NO (`/v1/audio/transcriptions` = `run_batch.cpp:188` residual) | NO | tests-only reachability | | Whisper audio encoder | NO | `whisper_audio.cpp:174` | NO | NO | tests-only callers | | Pooling / embeddings | NO (`is_pooling_model=false` in all 27) | `layers/pooler/*.cpp`, `pool/pooling_runner` (`ENG-POOLER-SEQ`) | NO (`/v1/embeddings` = residual) | NO | `PoolingRunner` test-only | @@ -75,11 +75,13 @@ JSON sub-config, not a separate surface) β€” the model for how the others should Public boundary = `#include "vllm.h"` only. 13 example units; `examples/cli` (vllm-cli, links `vllm::shared`, `#include "vllm.h"` only, `cli/main.cpp:16`) is the sole clean ABI -client. The other 12 reach `include/vllm/**` / `vt/**` and are transition-tracked in -`scripts/example-abi-allowlist.txt`: +client. **ROW 1 UPDATE (2026-08-07): `parakeet_transcribe` is the SECOND clean ABI +client** β€” the Parakeet fold rewrote it against `vllm.h` + `vllm::shared` only, and the +ratchet fell 12 -> 11. The remaining 11 reach `include/vllm/**` / `vt/**` and are +transition-tracked in `scripts/example-abi-allowlist.txt`: - Capability drivers: `deepseek_v4_gen`, `laguna_gen`, `kimi_linear_gen`, - `minimax_h3_gen`, `minimax_h3_mux`, `parakeet_transcribe`, `server`. + `minimax_h3_gen`, `minimax_h3_mux`, `server`. - Dev/diagnostic (internal-by-nature, folded for consistency): `bench` (via `bench_core.h`), `tokenize`, `dump_container`, `dequant_nvfp4`, `quant_gemm_bench`. - Out of the gated `examples/` tree: `benchmarks/vulkan_gemm_ab.cpp` (Vulkan A/B harness). @@ -87,7 +89,7 @@ client. The other 12 reach `include/vllm/**` / `vt/**` and are transition-tracke **Policy (developer-directed 2026-08-07): no permanent exemptions.** Every allowlist entry β€” drivers AND dev/diagnostic tools β€” is a transition-tracker pointing at a fold row; the guard fails on any internal include not tracked, and a shrink-only ratchet -(`MAX_INTERNAL_REACHING = 12`) means the count can only fall as folds land, never grow to +(`MAX_INTERNAL_REACHING`, 11 since ROW 1) means the count can only fall as folds land, never grow to admit a new violation. The public header set is DERIVED from the CMake install rules (exactly `include/vllm.h` today), not hardcoded. The guard catches BOTH breach vectors: a `#include "vllm/..."|"vt/..."|"src/..."` AND a CMake `-I` grant into the internal tree @@ -129,7 +131,7 @@ lanes are leaves of `ARCH-ONE-SURFACE` (do not open parallel rows). | 1 | Video+audio gen | `vllm_video_generate` + job/status/content entry points (mirror `/v1/videos`; carry the `VideoRunner` internally, not example-injected) | rewrite `minimax_h3_gen`/`server` as clients; delete driver glue | L | H3 loaders; ffmpeg-mux boundary (ratified in `examples/`) | | 2 | Laguna fast decode | make the registered `LagunaForCausalLM` forward route the keep-quant/NVFP4 device path (retire the stub); load keep-quant GGUF/NVFP4 dirs through `vllm_engine_load` | rewrite `laguna_gen`; delete `LagunaForwardGguf*` | M | keep-quant load in the engine loader | | 3 | DeepSeek-V4 fast decode | same as (2) for `DeepseekV4ForCausalLM`; real MLA paged KV (retire the W3 stub) | rewrite `deepseek_v4_gen`; delete `DeepseekV4ForwardGguf*` | M | MLA paged-KV topology | -| 4 | Audio transcription | `vllm_transcribe` + live `/v1/audio/transcriptions`; register a Whisper/Voxtral/Parakeet arch or an encoder lane | rewrite `parakeet_transcribe`; wire the route | M | encoderβ†’text seam | +| 4 | Audio transcription | **DONE (ROW 1, 2026-08-07)**: `vllm_transcribe` (ABI v11) + live `/v1/audio/transcriptions`; ParakeetForCTC/RNNT/TDT registered (SupportsTranscription mirror, refuse-by-task) | **DONE**: `parakeet_transcribe` rewritten as a `vllm.h` client (byte-identical transcript goldens); route live, task-conditional | M | encoderβ†’text seam (LANDED: `ParakeetTranscriber`) | | 5 | Kimi-Linear incremental | expose the incremental decode path through the runner/engine (the recompute forward already routes) | rewrite `kimi_linear_gen` | S–M | `KimiDecodeCache` on the runner | | 6 | Embeddings/pooling | `vllm_embed`/pooling entry point + live `/v1/embeddings`; register a pooling arch (`is_pooling_model=true`); invoke `PoolingRunner` in the step | β€” | M | pooler live-wiring | | 7 | Multimodal input | multimodal-content entry point on `vllm_chat`; run the vision/audio tower in the engine step (`mm_features`β†’`ModelForwardInput.mm`) | wire `chat_mm` seam into the ABI | L | `MM-SERVE-E2E` engine mm-forward residual | diff --git a/.agents/state.md b/.agents/state.md index 55b5ea606..2f9110d1c 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -42031,3 +42031,71 @@ leaves (Kimi runner fold #279, Parakeet ASR #280). Reviewer findings 1-8 applied (quoted+angle includes, comment-stripped ABI tokenize, subprocess enforcement tests, ratchet equality pin, reachable-row-removal design note, meta-gap note). No CUDA build; no perf number owed; STATUS inside its char ratchet. + +## 2026-08-07 β€” ARCH-ONE-SURFACE ROW 1: Parakeet ASR folded onto the ONE surface (PR #121) + + +**Landed** (`row/PARAKEET-ONE-SURFACE`, helper claim = PR #121, base `f98e1e48`): +the first remediation slice of the ONE SURFACE program β€” audio transcription is +now reachable by every consumer, gated BYTE-IDENTICAL to the pre-fold example. + +**Correctness anchor first.** Real `nvidia/parakeet-*` checkpoints were deleted +post-#89, so a deterministic tiny CTC+RNNT fixture pair + clip was committed +(`tests/vllm/models/fixtures/parakeet_e2e`, generator +`scripts/mm/parakeet_e2e_fixture_gen.py`) and the transcript goldens were +captured from the PRE-refactor binary at `main@f98e1e48` BEFORE any change +(ctc: ids `3 4 3` / "atheat"; rnnt: 20 ids / "sss on on onssssss ..."). +`tests/vllm/models/test_parakeet_transcription_fold.cpp` holds seam == +replicated old pipeline == committed goldens, per head. + +**The fold, in the binding order.** +- W1a tokenizer: `Tokenizer::FromHfJson` accepts Metaspace `split:true` + (MergedWithNext pre-split; merges cannot cross ▁) and a bare `Metaspace` + DECODER node selects HF's decode_chain rule (first token drops ▁) β€” + the guard at `tokenizer.cpp:554` implemented instead of refusing; + Mistral/Gemma Sequence chain regression-pinned + (`tests/vllm/test_tokenizer_metaspace_split.cpp`, metaspace.rs ports). +- W1b/c library: `vllm::multimodal::ParakeetTranscriber` + (`parakeet_transcription.{h,cpp}`) absorbs WAV ingest + (`DecodeWavPcm16Mono`), model_type head dispatch and tokenizer decode; the + example's private `ReadWav16BitMono`/`LoadVocab`/`DecodeIds` are DELETED. +- W2 registry: `ParakeetForCTC/ForRNNT/ForTDT` registered + (`parakeet_registry.cpp`), `ModelInfo` grows the SupportsTranscription + mirror (interfaces.py:1110-1118); BEYOND-PIN breadth recorded (pinned vLLM: + Parakeet = NemotronH audio component only, registry.py:511-513). + Refuse-by-task: every factory hook + `LoadedEngine::FromModelDir` (via new + non-throwing `PeekHfArchitectures`) refuse actionably; registry pins moved + 30 -> 33. +- W3 C ABI: `vllm_transcribe` + params/result structs, ABI v10 -> 11; + `vllm_engine_load` task-dispatches a transcription-only dir to the seam; + text entrypoints and `vllm_transcribe` refuse each other's handles cleanly. + FIRST real-checkpoint load gated through the public ABI (`test_capi` v11 + cases close the audit's severity note). FEATURES abi-capability row flips + to reachable; `abi-capability-allowlist.txt` shrinks by the transcription + row. +- W4 server: task-conditional `/v1/audio/transcriptions` (multipart `file`, + response_format json/text) through the SAME seam; serving-less `ApiServer` + ctor = vLLM's supported_tasks-conditional registration (generate routes 404 + on an ASR server); `examples/server` dispatches by resolved arch; verified + LIVE with curl. Residuals: run_batch transcription line; + verbose_json/srt/vtt. +- W5 example: `examples/parakeet_transcribe` = thin `vllm.h` client + (`vllm::shared`), stdout DIFF-IDENTICAL to the pre-fold binary on both + fixtures; `example-abi-allowlist.txt` loses its row; + `MAX_INTERNAL_REACHING` 12 -> 11 with the test pin + spec claims moved. + +**Gates.** Full CPU build clean (`-Werror`, 0 warnings); ctest 349/349 (345 +parallel + 4 serial; the only 2 reds were the registry count pins, moved as +their comments instruct); surface guard + 46-case mutation suite green with +the SHRINK enforced; every new test mutation-verified red/green (tokenizer +decode rule + split gate, CTC collapse, transcription-only flag, ABI dispatch ++ input validation, server response_format + route gate). + +**Residuals (honest).** (1) run_batch `/v1/audio/transcriptions` stays a named +residual (batch line shape carries no audio). (2) verbose_json/srt/vtt +response formats -> 400 naming the residual. (3) A REAL pretrained checkpoint +was not re-downloaded (safe defaults); the fold gate rests on the committed +synthetic pair + the P4/P6 pretrained evidence. (4) A real Parakeet +tokenizer.json may carry normalizer fields `FromHfJson` refuses loudly β€” +untestable CPU-side without the checkpoint, fails loud not wrong. (5) Whisper/ +Voxtral remain off-registry (fold #9/#10 of the audit). diff --git a/CMakeLists.txt b/CMakeLists.txt index 66d95f2a8..b5168a919 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -666,6 +666,13 @@ add_library(vllm STATIC src/vllm/model_executor/models/parakeet_transducer.cpp src/vllm/model_executor/models/parakeet_weights.cpp src/vllm/multimodal/parakeet_audio_processor.cpp + # The ONE transcription seam every consumer drives (C ABI vllm_transcribe, + # the server's /v1/audio/transcriptions, the parakeet-transcribe example) β€” + # ARCH-ONE-SURFACE ROW 1: absorbs the pipeline examples/parakeet_transcribe + # used to own privately. The refuse-by-task registry TU registers the three + # Parakeet head archs so config.json RESOLVES (SupportsTranscription mirror). + src/vllm/multimodal/parakeet_transcription.cpp + src/vllm/model_executor/models/parakeet_registry.cpp src/vllm/v1/core/encoder_cache_manager.cpp src/vllm/tokenizer/unicode_data.cpp src/vllm/tokenizer/pretokenizer.cpp diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 329bf008f..bfe286491 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -336,7 +336,7 @@ built on it rather than keeping the flattering one. | Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: nothing measured, claimed or owed.** 16 NATIVE kernels; 71 on the host tier. opt-125m runs e2e token-exact on llvmpipe; no speed measured. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** The W0 skeleton registers 1 of 106 ops and its HIP sources have never been compiled by anyone; no AMD hardware here | A contributor's first `-DVLLM_CPP_HIP=ON` build ([#41](https://github.com/mudler/vllm.cpp/issues/41)). Only once a model runs does a same-box vLLM-ROCm oracle become the gate; the floor is vLLM, quant-matched | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | -| Parakeet/FastConformer ASR (kernels P1-P3 + encoder/CTC P4) | **NO number measured, claimed or owed.** Correctness-gated only, CPU f32; no CUDA provider, no GPU suite run, no aarch64 run. | Floor is `parakeet.cpp`, same clip and box; needs a CUDA provider and a pretrained checkpoint | +| Parakeet/FastConformer ASR (P1-P4 + ONE-SURFACE fold ROW 1) | **NO number measured, claimed or owed.** Correctness-gated only, CPU f32; the 2026-08-07 surface fold (`vllm_transcribe`, `/v1/audio/transcriptions`) is transcript-byte-identical plumbing, no speed claim. | Floor is `parakeet.cpp`, same clip and box; needs a CUDA provider and a pretrained checkpoint | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | | Ampere consumer (`sm_86`, RTX 3090 class) | **No number owed; no such board here.** 2026-08-06 build-verify: 7/7 FA2 TUs 0-warn, real `sm_86` SASS. [Detail](../.agents/benchmark-record.md) | External RTX 3090 report. Floor is llama.cpp on that card (GGUF, not our Blackwell-only NVFP4 grid) | | Pre-Ampere breadth (Turing `sm_75` / Volta `sm_70` / Pascal) | **No number owed; nothing runs on these arches.** 2026-08-06 `sm_75`: 20/20 TUs PASS (0 err/warn), WMMA bodies + all 3 selectors arch-gated; GB10 SASS byte-identical. [Detail](../.agents/benchmark-record.md) | Full-library LINK at `sm_75` + `cuobjdump` SASS, then a build-supported row. The fp16 `fattn` port is speed-only now; its floor when a card exists is llama.cpp on that card | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 0fcfcd370..48964fb76 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -121,6 +121,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Paged-incremental decode (Β§19) GB10: 18.9 tok/s (0.90Γ— vLLM) @ 122/128 = coherent best**; STRICT unreachable (bf16 stream REFUTED Β§20/#118 122β†’4/128; p7 intrinsic near-tie) | CLI opt-in `--incremental`; SERVER fold scoped (ARCH-ONE-SURFACE req 4; runner aborts on Kimi KV) | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | +| `ParakeetForCTC`, `ParakeetForRNNT`, `ParakeetForTDT` | nvidia/parakeet-ctc-0.6b/-1.1b, -rnnt-0.6b, -tdt-0.6b-v3 (transcribed, ids exact vs HF `generate()`, P4/P6 2026-08-07; not retained) + committed synthetic fold fixture | ASR transcription-only (`SupportsTranscription` mirror; text paths refuse by task); fold gate byte-identical to the pre-refactor pipeline | n/a (CPU correctness-grade ASR via `vllm_transcribe` + `/v1/audio/transcriptions`) | | `CohereForCausalLM` | Command-R / Cohere (and Cohere2) | scaffold: W0 tiny-random oracle run-verified; real-checkpoint gate blocked | no run | @@ -243,7 +244,7 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Speculative decoding config | `speculative_config` | reachable | | Custom logits processor | `vllm_logits_processor` | reachable | | Embeddings / pooling | none | embedder-unreachable | -| Audio transcription | none | embedder-unreachable | +| Audio transcription (Parakeet ASR) | `vllm_transcribe`, `vllm_transcription_params_default`, `vllm_transcription_free` | reachable | | Video+audio generation (MiniMax-H3) | none | embedder-unreachable | | Multimodal input (image/audio/video) | none | embedder-unreachable | diff --git a/docs/STATUS.md b/docs/STATUS.md index faef3f10d..44928d556 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -405,7 +405,7 @@ re-read the whole weight 131 times), and an opt-in load-time `[N,K]` to `[K,N]` Byte-identical: both orientations accumulate each output over K in strict increasing order, so the repack is a layout choice and never a numerical one. -Parakeet/FastConformer ASR (2026-08-07): *correctness-complete, CPU only*. Kernels, encoder, CTC greedy, log-mel; ids exact vs HF `ParakeetForCTC` oracle; transcribes ctc-0.6b/1.1b, rnnt-0.6b, tdt-0.6b-v3. No CUDA/aarch64, no speed number. +Parakeet ASR (2026-08-07): *CPU-correct, ON THE ONE SURFACE (ROW 1)*. Ids exact vs HF; registered transcription-only; `vllm_transcribe` (ABI v11) + `/v1/audio/transcriptions`; thin `vllm.h` example, golden-exact; ratchet 12 -> 11. No CUDA, no speed number. ## Not supported yet diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 692c99eee..f6d7e14dc 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -99,11 +99,11 @@ if(VLLM_CPP_VULKAN) vllm_cpp_set_warnings(vulkan-dispatch-floor) endif() -# parakeet-transcribe: the ASR driver β€” real WAV, real pretrained -# ParakeetForCTC weights, log-mel -> encoder -> CTC -> collapsed token ids. -# Lives here for the same reason the h3 drivers do: it is a driver over real -# checkpoints, not library surface. The unit gates prove the math against a -# random-weight HF oracle; this proves the model actually transcribes. +# parakeet-transcribe: the ASR example β€” a THIN client of the public C ABI +# (include/vllm.h + vllm::shared ONLY, like vllm-cli), per the ONE SURFACE +# directive. The pipeline it used to own privately lives in the library behind +# vllm_engine_load + vllm_transcribe (ARCH-ONE-SURFACE ROW 1); the fold gate +# holds its transcript byte-identical to the pre-fold binary. add_executable(parakeet-transcribe parakeet_transcribe/main.cpp) -target_link_libraries(parakeet-transcribe PRIVATE vllm::vllm) +target_link_libraries(parakeet-transcribe PRIVATE vllm::shared) vllm_cpp_set_warnings(parakeet-transcribe) diff --git a/examples/parakeet_transcribe/main.cpp b/examples/parakeet_transcribe/main.cpp index 743757908..75a8b8695 100644 --- a/examples/parakeet_transcribe/main.cpp +++ b/examples/parakeet_transcribe/main.cpp @@ -1,264 +1,84 @@ -// Parakeet ASR end to end on real audio, real pretrained weights, CPU only, -// for the WHOLE published model family, not just CTC. +// Parakeet ASR end to end β€” a THIN CLIENT of the public C ABI (include/vllm.h) +// and NOTHING else, per the ONE SURFACE directive (ARCH-ONE-SURFACE ROW 1). // -// This exists to answer one question the unit gates deliberately do NOT answer: -// does the port actually TRANSCRIBE? The oracles in -// tests/vllm/models/test_parakeet_ctc_engine.cpp and -// tests/vllm/models/test_parakeet_transducer.cpp gate our forward against real -// HF modules, but with RANDOM weights, so they prove the math and nothing about -// speech. Neither produces a transcript, on purpose. +// The pre-fold version of this example owned the whole pipeline privately +// (its own WAV reader, the model_type head dispatch, a LoadVocab + DecodeIds +// pair working around the tokenizer's Metaspace split=true refusal). All of +// that now lives in the library behind vllm_engine_load + vllm_transcribe β€” +// the SAME entry points any embedder gets β€” and this file keeps exactly what +// an example may own: argv parsing, timing, and printing. The transcript is +// byte-identical to the pre-fold binary (gated by +// tests/vllm/models/test_parakeet_transcription_fold.cpp and the committed +// goldens in tests/vllm/models/fixtures/parakeet_e2e). // -// So: 16 kHz mono WAV -> log-mel (ParakeetAudioProcessor) -> encoder -> the head -// config.json names -> token ids -> text. +// parakeet-transcribe [reps] // -// parakeet-transcribe -// -// `` is any HF-format Parakeet checkpoint; the head is taken from -// `model_type` in config.json, so one binary covers all three: -// parakeet_ctc -> ParakeetForCTC (nvidia/parakeet-ctc-0.6b, -ctc-1.1b) -// parakeet_rnnt -> ParakeetForRNNT (nvidia/parakeet-rnnt-0.6b, -rnnt-1.1b) -// parakeet_tdt -> ParakeetForTDT (nvidia/parakeet-tdt-0.6b-v3) -// -// Token ids always go to stdout, diffable against HF's own `generate()`. When -// the checkpoint ships a `tokenizer.json` the decoded TEXT is printed too; see -// the decoder note on `DecodeIds` for exactly which upstream rule that mirrors -// and why it does not reuse vllm::Tokenizer. +// `` is any HF-format Parakeet checkpoint (CTC / RNN-T / +// TDT β€” the head comes from config.json, resolved inside the library). Token +// ids always go to stdout, diffable against HF's own `generate()`; when the +// checkpoint ships a tokenizer.json the decoded TEXT is printed too. The +// optional `reps` repeats the TRANSCRIPTION call so the checkpoint load stays +// out of the timed region (the load happens once, in vllm_engine_load). #include -#include #include #include -#include -#include -#include #include -#include - -#include - -#include "vllm/model_executor/models/parakeet_encoder.h" -#include "vllm/model_executor/models/parakeet_transducer.h" -#include "vllm/multimodal/parakeet_audio_processor.h" -#include "vt/backend.h" - -namespace { - -// Minimal RIFF/WAVE reader: 16-bit PCM mono, which is what the LibriSpeech and -// NeMo sample clips are. Anything else is refused loudly rather than resampled, -// mirroring the extractor's own refusal to resample -// (feature_extraction_parakeet.py:195-201). -bool ReadWav16BitMono(const std::string& path, std::vector* out, int* sample_rate) { - std::ifstream f(path, std::ios::binary); - if (!f) { - std::fprintf(stderr, "cannot open %s\n", path.c_str()); - return false; - } - char riff[12]; - f.read(riff, 12); - if (std::memcmp(riff, "RIFF", 4) != 0 || std::memcmp(riff + 8, "WAVE", 4) != 0) { - std::fprintf(stderr, "%s is not a RIFF/WAVE file\n", path.c_str()); - return false; - } - int channels = 0; - int bits = 0; - while (f) { - char id[4]; - uint32_t sz = 0; - f.read(id, 4); - f.read(reinterpret_cast(&sz), 4); - if (!f) break; - if (std::memcmp(id, "fmt ", 4) == 0) { - std::vector fmt(sz); - f.read(fmt.data(), sz); - uint16_t ch = 0, bps = 0; - uint32_t sr = 0; - std::memcpy(&ch, fmt.data() + 2, 2); - std::memcpy(&sr, fmt.data() + 4, 4); - std::memcpy(&bps, fmt.data() + 14, 2); - channels = ch; - bits = bps; - *sample_rate = static_cast(sr); - } else if (std::memcmp(id, "data", 4) == 0) { - if (channels != 1 || bits != 16) { - std::fprintf(stderr, "need 16-bit mono, got %d-bit %d-channel\n", bits, channels); - return false; - } - const size_t n = sz / 2; - std::vector pcm(n); - f.read(reinterpret_cast(pcm.data()), sz); - out->resize(n); - for (size_t i = 0; i < n; ++i) { - (*out)[i] = static_cast(pcm[i]) / 32768.0F; - } - return true; - } else { - f.seekg(sz, std::ios::cur); - } - } - std::fprintf(stderr, "%s has no data chunk\n", path.c_str()); - return false; -} - -// id -> piece from a `tokenizer.json`, taking both the BPE vocab and the -// `added_tokens` (which is where `` / `` / the multilingual prompt -// tokens live). Empty when the checkpoint ships no tokenizer. -std::map LoadVocab(const std::string& dir) { - std::map vocab; - std::ifstream f(dir + "/tokenizer.json", std::ios::binary); - if (!f.good()) return vocab; - nlohmann::json doc; - f >> doc; - const auto model = doc.find("model"); - if (model != doc.end()) { - const auto v = model->find("vocab"); - if (v != model->end() && v->is_object()) { - for (auto it = v->begin(); it != v->end(); ++it) { - vocab[it.value().get()] = it.key(); - } - } - } - const auto added = doc.find("added_tokens"); - if (added != doc.end() && added->is_array()) { - for (const auto& t : *added) { - vocab[t.at("id").get()] = t.at("content").get(); - } - } - return vocab; -} -// The `Metaspace` DECODER rule, which is what every published Parakeet -// tokenizer.json declares (`{"type": "Metaspace", "replacement": "▁", -// "prepend_scheme": "always", "split": true}`): inside the FIRST piece the -// replacement character is DROPPED, inside every later piece it becomes a space, -// and the pieces are concatenated. Mirrors HF tokenizers -// `decoders::metaspace::Metaspace::decode_chain`. -// -// This does not go through `vllm::Tokenizer` on purpose: `Tokenizer::FromHfJson` -// refuses a Metaspace pre_tokenizer with `split: true` ("no golden in scope", -// src/vllm/tokenizer/tokenizer.cpp), which is an ENCODE-side restriction. This -// example only ever decodes, so it implements the decoder rule directly rather -// than weakening that guard to get at it. -std::string DecodeIds(const std::vector& ids, - const std::map& vocab) { - static const std::string kReplacement = "\xe2\x96\x81"; // U+2581 LOWER ONE EIGHTH BLOCK - std::string text; - for (size_t i = 0; i < ids.size(); ++i) { - const auto it = vocab.find(ids[i]); - if (it == vocab.end()) continue; - const std::string& piece = it->second; - for (size_t p = 0; p < piece.size();) { - if (piece.compare(p, kReplacement.size(), kReplacement) == 0) { - if (i != 0) text.push_back(' '); - p += kReplacement.size(); - } else { - text.push_back(piece[p]); - ++p; - } - } - } - return text; -} - -void PrintIdsAndText(const std::vector& ids, - const std::map& vocab) { - for (size_t i = 0; i < ids.size(); ++i) { - std::printf("%s%d", i ? " " : "", ids[i]); - } - std::printf("\n"); - if (!vocab.empty()) { - std::printf("%s\n", DecodeIds(ids, vocab).c_str()); - } else { - std::fprintf(stderr, "no tokenizer.json in the checkpoint: ids only\n"); - } -} - -} // namespace +#include "vllm.h" int main(int argc, char** argv) { if (argc < 3) { - std::fprintf(stderr, "usage: %s \n", argv[0]); + std::fprintf(stderr, "usage: %s [reps]\n", + argv[0]); return 2; } const std::string ckpt = argv[1]; const std::string wav = argv[2]; + const int reps = (argc > 3) ? std::atoi(argv[3]) : 1; - std::vector samples; - int sample_rate = 0; - if (!ReadWav16BitMono(wav, &samples, &sample_rate)) return 1; - std::fprintf(stderr, "audio: %zu samples @ %d Hz (%.2f s)\n", samples.size(), sample_rate, - static_cast(samples.size()) / sample_rate); - - const std::string model_type = vllm::multimodal::LoadParakeetModelType(ckpt); - const vllm::multimodal::ParakeetEncoderConfig probe = - vllm::multimodal::LoadParakeetConfig(ckpt); - std::fprintf(stderr, "model_type: %s (%lld mel bins)\n", model_type.c_str(), - static_cast(probe.num_mel_bins)); - - // The extractor is driven by the checkpoint's own `num_mel_bins`: 80 on the - // CTC and RNN-T checkpoints, 128 on parakeet-tdt-0.6b-v3. - vllm::multimodal::ParakeetExtractorConfig ecfg; - ecfg.feature_size = static_cast(probe.num_mel_bins); - const vllm::multimodal::ParakeetAudioProcessor proc(ecfg); - const vllm::multimodal::ParakeetAudioFeatures feats = - proc.ProcessWaveform(samples.data(), static_cast(samples.size()), sample_rate); - std::fprintf(stderr, "features: %lld frames (%lld valid)\n", - static_cast(feats.num_frames), - static_cast(feats.valid_frames)); - - vt::Backend& cpu = vt::GetBackend(vt::DeviceType::kCPU); - const std::map vocab = LoadVocab(ckpt); - - if (model_type == "parakeet_rnnt" || model_type == "parakeet_tdt") { - vllm::multimodal::ParakeetEncoderConfig enc_cfg; - vllm::multimodal::ParakeetTransducerConfig cfg; - const vllm::multimodal::ParakeetForTransducerWeights w = - vllm::multimodal::LoadParakeetTransducer(ckpt, &enc_cfg, &cfg); - std::fprintf(stderr, "weights: %zu encoder layers, %lld decoder LSTM layers, vocab %lld%s\n", - w.encoder.layers.size(), static_cast(cfg.num_decoder_layers), - static_cast(cfg.vocab_size), - cfg.is_tdt() ? " (+ TDT duration head)" : ""); - - const vllm::multimodal::ParakeetTransducerOutput out = - vllm::multimodal::ParakeetForTransducerForward(feats.input_features, feats.num_frames, - feats.valid_frames, w, enc_cfg, cfg, cpu); - std::fprintf(stderr, "encoder: %lld frames (%lld valid); decode: %zu steps, %zu tokens\n", - static_cast(out.encoder_frames), - static_cast(out.valid_encoder_frames), - out.sequences.size() - 1, out.token_ids.size()); - PrintIdsAndText(out.token_ids, vocab); - return 0; - } - - if (model_type != "parakeet_ctc" && !model_type.empty()) { - std::fprintf(stderr, "unsupported model_type '%s'\n", model_type.c_str()); + vllm_model_params mp = vllm_model_params_default(); + mp.model_path = ckpt.c_str(); + vllm_engine* engine = nullptr; + if (vllm_engine_load(&mp, &engine) != VLLM_OK) { + std::fprintf(stderr, "load failed: %s\n", vllm_last_error()); return 1; } - vllm::multimodal::ParakeetEncoderConfig cfg; - const vllm::multimodal::ParakeetForCTCWeights w = - vllm::multimodal::LoadParakeetForCTC(ckpt, &cfg); - std::fprintf(stderr, "weights: %zu encoder layers, vocab %lld\n", w.encoder.layers.size(), - static_cast(cfg.vocab_size)); + vllm_transcription_params tp = vllm_transcription_params_default(); + tp.audio_path = wav.c_str(); - // Optional third argument: repeat the FORWARD only, so the checkpoint load and - // the mel front end stay OUT of the timed region. Without it the wall clock is - // dominated by reading a multi-GiB safetensors, and a 3 s clip can time SLOWER - // than a 10 s one: a loader measurement wearing a model's clothes. - const int reps = (argc > 3) ? std::atoi(argv[3]) : 1; - vllm::multimodal::ParakeetCTCOutput out; + vllm_transcription out; double best_ms = 0.0; + vllm_status st = VLLM_OK; for (int r = 0; r < reps; ++r) { + if (r > 0) vllm_transcription_free(&out); const auto t0 = std::chrono::steady_clock::now(); - out = vllm::multimodal::ParakeetForCTCForward( - feats.input_features, feats.num_frames, feats.valid_frames, w, cfg, cpu); + st = vllm_transcribe(engine, &tp, &out); const auto t1 = std::chrono::steady_clock::now(); - const double ms = std::chrono::duration(t1 - t0).count(); + if (st != VLLM_OK) break; + const double ms = + std::chrono::duration(t1 - t0).count(); if (r == 0 || ms < best_ms) best_ms = ms; - std::fprintf(stderr, "forward rep %d: %.1f ms\n", r, ms); + if (reps > 1) std::fprintf(stderr, "transcribe rep %d: %.1f ms\n", r, ms); + } + if (st != VLLM_OK) { + std::fprintf(stderr, "transcribe failed: %s\n", vllm_last_error()); + vllm_engine_free(engine); + return 1; + } + if (reps > 1) std::fprintf(stderr, "transcribe best: %.1f ms\n", best_ms); + + for (int32_t i = 0; i < out.n_token_ids; ++i) { + std::printf("%s%d", i ? " " : "", out.token_ids[i]); } - std::fprintf(stderr, "forward best: %.1f ms\n", best_ms); - std::fprintf(stderr, "encoder: %lld output frames (%lld valid)\n", - static_cast(out.num_output_frames), - static_cast(out.valid_output_frames)); - PrintIdsAndText(out.token_ids, vocab); + std::printf("\n"); + if (out.has_text) { + std::printf("%s\n", out.text); + } else { + std::fprintf(stderr, "no tokenizer.json in the checkpoint: ids only\n"); + } + + vllm_transcription_free(&out); + vllm_engine_free(engine); return 0; } diff --git a/examples/server/main.cpp b/examples/server/main.cpp index 9544e21df..0d0ab66c6 100644 --- a/examples/server/main.cpp +++ b/examples/server/main.cpp @@ -38,6 +38,7 @@ #include #include #include +#include #include #include #include @@ -68,6 +69,8 @@ #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/qwen3_5_weights.h" #include "vllm/transformers_utils/hf_config.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/multimodal/parakeet_transcription.h" #include "vllm/tokenizer/tokenizer.h" #include "vllm/version.h" #include "vllm/v1/core/kv_cache_utils.h" @@ -464,6 +467,50 @@ int main(int argc, char** argv) { : dir.parent_path().filename().string()) : args.served_model_name; + // ── TASK DISPATCH (ARCH-ONE-SURFACE ROW 1): a model dir whose + // architectures resolve to a SupportsTranscription-ONLY registration + // (Parakeet CTC/RNNT/TDT) serves /v1/audio/transcriptions through the ONE + // library seam β€” the same ParakeetTranscriber vllm_transcribe drives β€” and + // registers NO generate routes (vLLM's task-conditional registration, + // api_server.py:255-265). Every other model takes the text path below, + // byte-identical to before. ──────────────────────────────────────────────── + { + bool transcription_only = false; + const std::vector archs = + vllm::PeekHfArchitectures(config_path); + if (!archs.empty()) { + try { + transcription_only = + vllm::ModelRegistry::Resolve(std::span(archs)) + .info.supports_transcription_only; + } catch (const std::exception&) { + transcription_only = false; // unknown arch: the text path diagnoses + } + } + if (transcription_only) { + std::cerr << "server: transcription-only model (" << archs[0] + << "); serving /v1/audio/transcriptions\n"; + auto transcriber = + std::make_shared( + vllm::multimodal::ParakeetTranscriber::FromDir(args.model_dir)); + namespace oai = vllm::entrypoints::openai; + oai::OpenAIServingModels asr_models(served_model_name); + oai::ApiServer asr_server(asr_models, vllm::Version()); + asr_server.set_transcriber( + [transcriber](const uint8_t* wav, size_t n) { + return transcriber->TranscribeWavBytes(wav, n); + }); + std::cerr << "server: listening on http://" << args.host << ":" + << args.port << "\n"; + if (!asr_server.listen(args.host, args.port)) { + std::cerr << "server: failed to bind " << args.host << ":" + << args.port << "\n"; + return 1; + } + return 0; + } + } + // ── Load the model + build the full engine stack via the shared loader // (src/vllm/entrypoints/model_loader.cpp) β€” the same path the C ABI drives. // It loads config.json + tokenizer.json + *.safetensors and wires the M1.8 diff --git a/include/vllm.h b/include/vllm.h index 87cc18e9c..9cf643e2e 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -81,8 +81,17 @@ extern "C" { * when set, overrides this field. 0 (the default) is the byte-identical * default, so zero-filling the struct growth keeps a v9 engine byte-identical. * Scheduler policy (incl. SGLang's cache-aware LPM) is selected through the v9 - * string field .scheduling_policy = "lpm" β€” there is no separate int knob. */ -#define VLLM_ABI_VERSION 10 + * string field .scheduling_policy = "lpm" β€” there is no separate int knob. + * v11: AUDIO TRANSCRIPTION (ARCH-ONE-SURFACE fold #4) β€” vllm_transcribe / + * vllm_transcription_params(_default) / vllm_transcription(_free), appended so + * zero values preserve behaviour. vllm_engine_load now RESOLVES a + * transcription-only checkpoint (Parakeet CTC/RNNT/TDT, the vLLM + * SupportsTranscription mirror) into a transcription engine: the text entry + * points on such a handle report VLLM_ERR_INVALID_ARGUMENT with an actionable + * message instead of serving, and vllm_transcribe on a TEXT handle does the + * same. A pre-v11 caller that never loads a Parakeet directory is + * byte-identical. */ +#define VLLM_ABI_VERSION 11 /* ── Export macro ───────────────────────────────────────────────────────────── * Marks the symbols that make up the stable ABI. Default visibility now; Task 3 @@ -452,6 +461,60 @@ VLLM_API vllm_status vllm_chat_stream(vllm_engine* engine, const char* request_json, vllm_token_callback cb, void* user_data); +/* ── Audio transcription (ABI v11) ──────────────────────────────────────────── + * The transcription slice of the ONE-SURFACE fold: an engine loaded from a + * transcription-only checkpoint (a directory whose config.json architectures + * resolve to the Parakeet CTC / RNN-T / TDT family β€” the mirror of vLLM's + * SupportsTranscription protocol) transcribes audio to text through the SAME + * library pipeline the bundled server's /v1/audio/transcriptions route and the + * parakeet-transcribe example drive. Greedy decode, 16 kHz mono input (the + * extractor refuses to resample, mirroring the HF feature extractor). + * + * Input is EXACTLY ONE of: + * - audio_path: a 16-bit PCM mono RIFF/WAVE file path; + * - pcm + n_samples + sample_rate: a mono float32 waveform in [-1, 1). + * Setting both or neither is VLLM_ERR_INVALID_ARGUMENT. All pointers are + * borrowed for the duration of the call. */ +typedef struct vllm_transcription_params { + const char* audio_path; /* WAV path, or NULL when pcm is used. */ + const float* pcm; /* mono f32 samples, or NULL when audio_path is used. */ + int64_t n_samples; /* number of entries in pcm. */ + int32_t sample_rate; /* pcm sampling rate in Hz (16000). */ +} vllm_transcription_params; + +/* One transcription result. OWNERSHIP: the caller frees text + token_ids via + * vllm_transcription_free(out) (text may also go through vllm_string_free). + * - text: heap NUL-terminated transcript. NULL when has_text == 0 (the + * checkpoint ships no tokenizer.json: ids only, exactly like the + * pre-fold example). + * - token_ids / n_token_ids: the decoded token ids (CTC: after the greedy + * collapse; transducer: blanks and the start token dropped). + * - has_text: 0/1 β€” whether `text` was produced. */ +typedef struct vllm_transcription { + char* text; + int32_t* token_ids; + int32_t n_token_ids; + int32_t has_text; +} vllm_transcription; + +/* Zero-initialized params (no input selected β€” the caller sets exactly one). */ +VLLM_API vllm_transcription_params vllm_transcription_params_default(void); + +/* Run a blocking transcription on a transcription-capable engine handle, + * filling *out. Returns VLLM_OK on success; VLLM_ERR_INVALID_ARGUMENT for a + * text-generation handle (use the completion/chat entry points there) or a bad + * input selection; VLLM_ERR_RUNTIME when the audio cannot be read/decoded + * (non-PCM16-mono WAV, wrong sample rate) or the forward fails β€” matching + * vllm_complete's convention. On any non-OK status *out is zeroed and + * vllm_last_error() carries the detail. */ +VLLM_API vllm_status vllm_transcribe(vllm_engine* engine, + const vllm_transcription_params* params, + vllm_transcription* out); + +/* Free the owned members of a transcription result and zero the struct. The + * struct itself is caller storage. NULL is a no-op. */ +VLLM_API void vllm_transcription_free(vllm_transcription* out); + /* ── Memory helpers ─────────────────────────────────────────────────────────── * Free a heap string returned by the library. NULL is a no-op. */ VLLM_API void vllm_string_free(char* s); diff --git a/include/vllm/entrypoints/openai/api_server.h b/include/vllm/entrypoints/openai/api_server.h index 65f27cc15..745f18cc9 100644 --- a/include/vllm/entrypoints/openai/api_server.h +++ b/include/vllm/entrypoints/openai/api_server.h @@ -24,6 +24,7 @@ #define VLLM_ENTRYPOINTS_OPENAI_API_SERVER_H_ #include +#include #include #include #include @@ -35,6 +36,7 @@ #include "vllm/entrypoints/openai/serving_completion.h" #include "vllm/entrypoints/openai/serving_models.h" #include "vllm/entrypoints/openai/video_api.h" +#include "vllm/multimodal/parakeet_transcription.h" namespace vllm::tok { class Tokenizer; @@ -71,6 +73,18 @@ class ApiServer { size_t max_concurrent_streams = kDefaultMaxConcurrentStreams, HttpWorkerPoolMode worker_pool_mode = HttpWorkerPoolMode::kCapacityFixed); + // TASK-CONDITIONAL construction without the text-generation serving stack + // (ARCH-ONE-SURFACE ROW 1): a server for a SupportsTranscription-ONLY model + // (Parakeet) has no AsyncLLM to build OpenAIServingCompletion/Chat around, so + // /v1/completions and /v1/chat/completions are simply NOT REGISTERED (404), + // mirroring vLLM registering the generate routes only when "generate" is in + // supported_tasks (api_server.py:255-265). Attach the transcription seam via + // set_transcriber; /v1/models, /health, /version and the other utility + // routes behave as before. + ApiServer(OpenAIServingModels& models, std::string version, + size_t max_concurrent_streams = kDefaultMaxConcurrentStreams, + HttpWorkerPoolMode worker_pool_mode = + HttpWorkerPoolMode::kCapacityFixed); ~ApiServer(); ApiServer(const ApiServer&) = delete; @@ -112,6 +126,18 @@ class ApiServer { // POST /v1/videos/sync -> run to completion, return the .mp4 path // GET /v1/videos/{id} -> job status // GET /v1/videos/{id}/content -> the finished MP4 bytes (video/mp4) + // POST /v1/audio/transcriptions (ARCH-ONE-SURFACE ROW 1). Mirror of vLLM's + // speech_to_text/transcription/api_router.py:31 `create_transcriptions` + + // serving.py:29 OpenAIServingTranscription: multipart `file` upload (16-bit + // PCM mono WAV here β€” the extractor refuses anything else loudly), optional + // `response_format` = "json" (default, -> TranscriptionResponse + // {"text": ...}) or "text" (-> the raw transcript, text/plain). + // verbose_json / srt / vtt are NAMED RESIDUALS -> 400. Registered ONLY when + // a transcriber is attached (the video_runner precedent); the route lambda + // extracts the upload and calls this with the raw file bytes. + DispatchResult handle_audio_transcriptions( + const std::string& file_bytes, const std::string& response_format) const; + DispatchResult handle_videos(const std::string& request_body); DispatchResult handle_videos_sync(const std::string& request_body); DispatchResult handle_video_status(const std::string& job_id) const; @@ -168,6 +194,19 @@ class ApiServer { video_runner_ = std::move(runner); } + // Attach the transcription seam backing POST /v1/audio/transcriptions. + // ADDITIVE and OPT-IN like the video runner above: absent => route + // unregistered => 404, byte-identical to a server without ASR. The callback + // wraps the ONE library entry + // (vllm::multimodal::ParakeetTranscriber::TranscribeWavBytes) β€” the SAME + // seam vllm_transcribe drives β€” so HTTP and FFI cannot drift. + using TranscribeFn = + std::function; + void set_transcriber(TranscribeFn transcriber) { + transcriber_ = std::move(transcriber); + } + // Attach the tokenizer + max_model_len backing /tokenize and /detokenize // (non-owning; must outlive the server). void set_tokenizer(const vllm::tok::Tokenizer* tokenizer, @@ -218,14 +257,18 @@ class ApiServer { size_t http_worker_count() const; private: - OpenAIServingCompletion& completion_; - OpenAIServingChat& chat_; + // Null in the serving-less (transcription-only) construction: the generate + // routes are then not registered, and direct handler dispatch reports the + // NotImplementedError mirror (see handle_completions). + OpenAIServingCompletion* completion_ = nullptr; + OpenAIServingChat* chat_ = nullptr; OpenAIServingModels& models_; std::string version_; // Opt-in C8 backings (all nullptr/empty by default β†’ routes not registered). const v1::metrics::PrometheusStatLogger* metrics_ = nullptr; ::vllm::openai::VideoRunner video_runner_; + TranscribeFn transcriber_; mutable ::vllm::openai::VideoJobStore video_jobs_; // Background workers for the ASYNC endpoint. Joined in ~ApiServer, which is // why they are joinable threads and not detached: a detached worker would diff --git a/include/vllm/model_executor/models/model_registry.h b/include/vllm/model_executor/models/model_registry.h index dc23dd8c1..2eb305de6 100644 --- a/include/vllm/model_executor/models/model_registry.h +++ b/include/vllm/model_executor/models/model_registry.h @@ -55,6 +55,17 @@ struct ModelInfo { bool is_hybrid = false; bool has_inner_state = false; bool supports_multimodal = false; + // Mirror of the SupportsTranscription protocol + // (vllm/model_executor/models/interfaces.py:1110-1118): + // `supports_transcription` marks an ASR-capable arch; + // `supports_transcription_only` marks one with NO text-generation path + // (interfaces.py:1118 `supports_transcription_only: ClassVar[bool]`), which + // the entrypoints use to refuse-by-task: LoadedEngine::FromModelDir rejects + // such an arch with a message pointing at the transcription entry points + // (vllm_transcribe / /v1/audio/transcriptions), mirroring how vLLM excludes + // "generate" from supported_tasks for them. + bool supports_transcription = false; + bool supports_transcription_only = false; std::string_view score_type = "bi-encoder"; }; diff --git a/include/vllm/multimodal/parakeet_transcription.h b/include/vllm/multimodal/parakeet_transcription.h new file mode 100644 index 000000000..1fb7653ae --- /dev/null +++ b/include/vllm/multimodal/parakeet_transcription.h @@ -0,0 +1,90 @@ +// Parakeet ASR transcription seam β€” the ONE library entry point that composes +// parakeet_audio_processor -> parakeet_encoder -> the head config.json names +// (CTC / RNN-T / TDT) -> vllm::Tokenizer decode, and that every consumer +// (C ABI `vllm_transcribe`, the OpenAI server's /v1/audio/transcriptions, the +// `parakeet-transcribe` example) drives. +// +// ARCH-ONE-SURFACE ROW 1 (fold #4, audio transcription). This file ABSORBS the +// pipeline `examples/parakeet_transcribe/main.cpp` used to own privately +// (pre-refactor main.cpp:176-264 @ f98e1e48: WAV read, model_type dispatch, +// head forward, id->text). Per the ONE SURFACE directive the example keeps +// argv parsing and printing ONLY; the capability lives here, reachable by any +// embedder. The id->text step routes through vllm::Tokenizer (which now +// implements the Metaspace split=true decoder the example carried a private +// DecodeIds for), and the WAV ingest through the library's DecodeWavPcm16Mono. +// +// Upstream mirror shape: vLLM serves ASR through SupportsTranscription models +// behind vllm/entrypoints/speech_to_text/ (transcription/serving.py:29 +// OpenAIServingTranscription -> base/serving.py `_create_speech_to_text`); +// this seam is the C++ engine-side equivalent of that handler's model call for +// the Parakeet family, whose forwards live in parakeet_encoder.h / +// parakeet_transducer.h (HF-mirrored; vLLM delegates the encoder to HF, +// parakeet.py:37,62). Greedy only β€” upstream's whole supported surface +// (`_supported_generation_modes`, modeling_parakeet.py main:925). +// +// CPU-only correctness-grade forward, matching the P4/P6 rows it composes; no +// speed claim is made here (spike Β§ Gates: the family's speed gate is GB10). +#pragma once + +#include +#include +#include +#include +#include + +namespace vllm::multimodal { + +// One transcription result. `token_ids` is exactly what the head decode +// emitted (CTC: greedy ids after the collapse; transducer: the emitted ids +// with blanks and the start token dropped) β€” the ids the pre-refactor example +// printed. `text` is the vllm::Tokenizer decode of those ids; `has_text` is +// false when the checkpoint ships no tokenizer.json (ids only, as before). +struct ParakeetTranscription { + std::vector token_ids; + std::string text; + bool has_text = false; +}; + +// A loaded Parakeet checkpoint (any head) ready to transcribe. Construction +// loads config.json, dispatches on `model_type` ("parakeet_ctc" -> +// ParakeetForCTC; "parakeet_rnnt"/"parakeet_tdt" -> the transducer heads; +// anything else throws), loads the weights, and loads tokenizer.json through +// vllm::Tokenizer when the checkpoint ships one. Throws std::runtime_error +// naming the problem on any mismatch β€” never a silently wrong transcript. +class ParakeetTranscriber { + public: + // `dir` is an HF-format Parakeet checkpoint directory (config.json + + // model.safetensors [+ index] [+ tokenizer.json] [+ generation_config.json]). + static ParakeetTranscriber FromDir(const std::string& dir); + + ParakeetTranscriber(ParakeetTranscriber&&) noexcept; + ParakeetTranscriber& operator=(ParakeetTranscriber&&) noexcept; + ~ParakeetTranscriber(); + + // "parakeet_ctc", "parakeet_rnnt" or "parakeet_tdt". + const std::string& model_type() const; + // True when tokenizer.json was present and loaded (text will be produced). + bool has_tokenizer() const; + + // Transcribe a mono float32 waveform. `sample_rate` must match the + // extractor's rate (16 kHz): the extractor refuses to resample, mirroring + // feature_extraction_parakeet.py:195-201. + ParakeetTranscription Transcribe(const float* samples, int64_t num_samples, + int sample_rate) const; + + // Transcribe a canonical 16-bit PCM mono RIFF/WAVE byte buffer + // (DecodeWavPcm16Mono β€” the library WAV ingest the example used to + // duplicate as ReadWav16BitMono). + ParakeetTranscription TranscribeWavBytes(const uint8_t* wav_bytes, + size_t num_bytes) const; + + // Read `wav_path` and TranscribeWavBytes it. + ParakeetTranscription TranscribeWavFile(const std::string& wav_path) const; + + private: + ParakeetTranscriber(); + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace vllm::multimodal diff --git a/include/vllm/tokenizer/tokenizer.h b/include/vllm/tokenizer/tokenizer.h index ad2d55303..effcd3c95 100644 --- a/include/vllm/tokenizer/tokenizer.h +++ b/include/vllm/tokenizer/tokenizer.h @@ -103,11 +103,20 @@ class Tokenizer { bool IsSentencePiece() const { return family_ == Family::kSentencePiece; } // Applies this SentencePiece tokenizer's DECODER chain to a window of already - // resolved token strings [begin, end): HF tokenizers' Sequence decoder - // Replace(▁->space) -> ByteFallback (runs of "<0xNN>" -> raw bytes, invalid - // UTF-8 run -> one U+FFFD per byte) -> Fuse -> Strip(1 leading space). Shared - // by Decode() and the incremental detokenizer so both mirror HF exactly. - // Precondition: IsSentencePiece(). + // resolved token strings [begin, end). Two chains exist, selected from the + // tokenizer.json `decoder` node at load: + // - default (Mistral/Gemma, no `Metaspace` decoder node): HF tokenizers' + // Sequence decoder Replace(▁->space) -> ByteFallback (runs of "<0xNN>" -> + // raw bytes, invalid UTF-8 run -> one U+FFFD per byte) -> Fuse -> + // Strip(1 leading space); + // - a bare `Metaspace` decoder node (every published Parakeet checkpoint): + // HF tokenizers 0.22 decoders `Metaspace::decode_chain` β€” inside the + // FIRST token of the window each replacement is DROPPED (unless + // prepend_scheme == "never", where it maps to a space like everywhere + // else), inside every later token it becomes ONE space; no ByteFallback, + // no Fuse, no Strip. + // Shared by Decode() and the incremental detokenizer so both mirror HF + // exactly. Precondition: IsSentencePiece(). std::string SpDecodeTokens(const std::vector& tokens, size_t begin, size_t end) const; @@ -139,6 +148,13 @@ class Tokenizer { std::string metaspace_replacement_; // e.g. "▁" (U+2581), UTF-8 PrependScheme prepend_scheme_ = PrependScheme::kFirst; bool metaspace_split_ = false; // Metaspace `split` flag + // --- DECODER selection (tokenizer.json `decoder` node). False => the + // Sequence chain (Replace -> ByteFallback -> Fuse -> Strip). True => the bare + // Metaspace decode_chain (Parakeet), with the DECODER node's own replacement + // and prepend_scheme (HF reads them off the decoder, not the pre_tokenizer). + bool sp_decoder_metaspace_ = false; + std::string sp_decoder_replacement_; + bool sp_decoder_prepend_never_ = false; bool byte_fallback_ = false; // model.byte_fallback bool fuse_unk_ = false; // model.fuse_unk (fuse consecutive unk ids) int32_t unk_id_ = -1; // model.unk_token resolved to an id, else -1 diff --git a/include/vllm/transformers_utils/hf_config.h b/include/vllm/transformers_utils/hf_config.h index af06e1728..3f169c6df 100644 --- a/include/vllm/transformers_utils/hf_config.h +++ b/include/vllm/transformers_utils/hf_config.h @@ -115,4 +115,13 @@ struct HfConfig { // path) on missing file, malformed JSON, or missing required fields. HfConfig LoadHfConfig(const std::string& path); +// Cheap, non-throwing peek at config.json's `architectures` array β€” empty on +// any parse/read problem. Exists for TASK dispatch BEFORE the full text-model +// HfConfig parse: a SupportsTranscription-only checkpoint (Parakeet) nests its +// fields under `encoder_config`, so LoadHfConfig's required-field errors would +// fire before the registry's refuse-by-task message could (ARCH-ONE-SURFACE +// ROW 1). Callers that need the parsed config still go through LoadHfConfig; +// this never replaces it. +std::vector PeekHfArchitectures(const std::string& path); + } // namespace vllm diff --git a/scripts/abi-capability-allowlist.txt b/scripts/abi-capability-allowlist.txt index da91ce49b..c26279d12 100644 --- a/scripts/abi-capability-allowlist.txt +++ b/scripts/abi-capability-allowlist.txt @@ -20,6 +20,5 @@ # Format: ` | fold= | `. embeddings / pooling | fold=ARCH-ONE-SURFACE | Engine-side pooler landed (ENG-POOLER-SEQ: pooler/{dispatch_pooler,methods,heads,poolers}.cpp + pool/pooling_runner) but PoolingRunner is never invoked by the live step, no arch sets is_pooling_model, /v1/embeddings is a run_batch "does not support endpoint" residual, and vllm.h has no embed/pool symbol. Fold: grow ABI pooling entry point, wire the live route, register a pooling arch -audio transcription | fold=ARCH-ONE-SURFACE | Whisper/Voxtral audio forwards exist (whisper_audio.cpp, voxtral.cpp) but are non-registered lanes called only from tests; /v1/audio/transcriptions is a run_batch residual, and vllm.h has no transcription symbol. Fold: grow ABI transcription entry point, wire the live route video+audio generation (minimax-h3) | fold=ARCH-ONE-SURFACE | The H3 diffusion pipeline serves over /v1/videos ONLY via an example-injected VideoRunner (set_video_runner, api_server.h:167; the library never sets it); vllm.h has no video symbol. Fold: the ARCH-ONE-SURFACE video ABI slice (order-of-work item 3), then rewrite minimax-h3-gen/server as clients multimodal input (image/audio/video) | fold=ARCH-ONE-SURFACE | vllm_chat/vllm_chat_stream are text-only (vllm.h:440,451); the C-ABI's EnsureChatServing never sets the mm seam (src/capi/vllm_c.cpp), and even the server's Qwen3-VL image seam does not run the vision tower in the live engine step (MM-SERVE-E2E residual). Fold: grow ABI multimodal-input entry point + the engine mm-forward diff --git a/scripts/check-supported-models.py b/scripts/check-supported-models.py index 6713811f8..9fd9c4429 100644 --- a/scripts/check-supported-models.py +++ b/scripts/check-supported-models.py @@ -58,7 +58,7 @@ # future registered arch stops matching this, the self-check below fails loudly # rather than silently dropping it from the comparison. ARCH_TOKEN_RE = re.compile( - r"`([A-Za-z0-9_]+For(?:CausalLM|ConditionalGeneration))`" + r"`([A-Za-z0-9_]+For(?:CausalLM|ConditionalGeneration|CTC|RNNT|TDT))`" ) diff --git a/scripts/check-surface-coverage.py b/scripts/check-surface-coverage.py index 9c69c5683..cc6e5f2b5 100644 --- a/scripts/check-surface-coverage.py +++ b/scripts/check-surface-coverage.py @@ -103,12 +103,14 @@ # as folds land (developer-directed 2026-08-07: no permanent exemptions, the allowlist # shrinks). A new internal-reacher cannot be added without consciously RAISING this # ceiling β€” a reviewable red flag, which is the point. Lower it as each example is folded -# onto the public ABI. 12 on 2026-08-07 (server + the 4 spec-named drivers + minimax_h3_mux -# + parakeet_transcribe + the 5 dev/diagnostic tools; examples/cli is the clean baseline). -# COUPLED: the "12" claims in .agents/specs/surface-coverage-2026-08-07.md and the state -# log, and the equality pin in tests/scripts/test_check_surface_coverage.py, must move in -# the SAME change as this constant. -MAX_INTERNAL_REACHING = 12 +# onto the public ABI. 11 since the ROW 1 Parakeet fold (2026-08-07: +# parakeet_transcribe became a clean vllm.h client; before it, 12 = server + the 4 +# spec-named drivers + minimax_h3_mux + parakeet_transcribe + the 5 dev/diagnostic +# tools; examples/cli is the clean baseline). +# COUPLED: the ratchet claims in .agents/specs/surface-coverage-2026-08-07.md and the +# state log, and the equality pin in tests/scripts/test_check_surface_coverage.py, must +# move in the SAME change as this constant. +MAX_INTERNAL_REACHING = 11 # An include INTO the internal C++ tree: `#include "vllm/..."` (the engine/model/layer # headers under include/vllm/, NOT the flat public "vllm.h"), `#include "vt/..."` (the diff --git a/scripts/example-abi-allowlist.txt b/scripts/example-abi-allowlist.txt index 03e559e67..c77161639 100644 --- a/scripts/example-abi-allowlist.txt +++ b/scripts/example-abi-allowlist.txt @@ -25,7 +25,6 @@ examples/deepseek_v4_gen | fold=ARCH-ONE-SURFACE | DeepSeek-V4-Flash keep-quant GGUF greedy decode (DeepseekV4ForwardGguf(Cached) + DeepseekV4KvCache) is CLI-only; the registered DeepseekV4ForCausalLM forward is a W3-W8 stub (deepseek_v4_registry.cpp:22). Grow ABI keep-quant GGUF load+decode, rewrite as ABI client, delete the bespoke forward examples/laguna_gen | fold=ARCH-ONE-SURFACE | Laguna-S-2.1 keep-quant GGUF (multi-shard) + NVFP4 W4A4 device-resident decode (LagunaForwardGguf(Cached), Marlin residents, fp4-shared) is CLI-only; registered LagunaForCausalLM forward VT_CHECK(false)s on non-bf16 (laguna.cpp:156). Grow ABI, rewrite as client, delete bespoke forward examples/kimi_linear_gen | fold=ARCH-ONE-SURFACE | Kimi-Linear-48B bf16-resident streaming loader + Β§18/Β§19 paged-incremental decode (ForwardPrefillIncremental/ForwardDecodeStepIncremental/KimiDecodeCache) is CLI-only (the recompute forward IS shared). Grow ABI incremental-decode path, rewrite as client, delete -examples/parakeet_transcribe | fold=ARCH-ONE-SURFACE | Parakeet/FastConformer ASR (audio transcription) runs only here (parakeet_encoder/parakeet_transducer + parakeet_audio_processor); Parakeet is off-registry (no REGISTER_VLLM_MODEL) and there is no transcription C-ABI. Grow the ABI transcription entry point, rewrite as client, delete the driver examples/minimax_h3_gen | fold=ARCH-ONE-SURFACE | The ENTIRE MiniMax-H3 video+audio diffusion pipeline (DiT + text encoder + VAEs + fl2va/ref2va + vision tower + ffmpeg) is CLI-only; MiniMaxH3 is off-registry (no REGISTER_VLLM_MODEL). Grow the video-gen ABI slice (ARCH-ONE-SURFACE order-of-work), rewrite as client, delete the driver glue examples/minimax_h3_mux | fold=ARCH-ONE-SURFACE | H3 PPM+WAV->MP4 ffmpeg mux helper (MiniMaxH3BuildMp4MuxArgs + fork/execvp) reaches internal minimax_h3 headers; the video-gen ABI slice should expose the assembled artifact so the mux can be a client (or fold into the video ABI). Grow ABI, rewrite, delete examples/server | fold=ARCH-ONE-SURFACE | The reference OpenAI server constructs the engine, metrics, video_runner and mm seam directly from internal C++ headers (LoadedEngine/AsyncLLM, minimax_h3.h, chat_mm). It should stand on the public surface (C ABI, or a curated public C++ API the ABI wraps). Grow the surface, rewrite the server against it diff --git a/scripts/mm/parakeet_e2e_fixture_gen.py b/scripts/mm/parakeet_e2e_fixture_gen.py new file mode 100644 index 000000000..27bfd4942 --- /dev/null +++ b/scripts/mm/parakeet_e2e_fixture_gen.py @@ -0,0 +1,317 @@ +#!/usr/bin/env python3 +"""Deterministic tiny Parakeet checkpoints + clip for the ONE-SURFACE fold gate. + +Row `ARCH-ONE-SURFACE` (Parakeet ASR fold): the refactor that moves the +`examples/parakeet_transcribe` pipeline into the library must reproduce the +EXACT ids + transcript the PRE-refactor example produced. The real +`nvidia/parakeet-*` checkpoints were deleted after PR #89 and AGENTS.md's safe +defaults forbid re-downloading multi-GB assets, so the gate runs on the +smallest checkpoint that exercises every stage: this generator emits + + tests/vllm/models/fixtures/parakeet_e2e/ + audio.wav deterministic 16 kHz mono PCM16 clip (0.25 s) + ctc/ HF-format ParakeetForCTC (config+safetensors+tokenizer) + rnnt/ HF-format ParakeetForRNNT (config+safetensors+tokenizer + +generation_config) + golden_ctc.txt stdout of the PRE-refactor example on ctc/ + audio.wav + golden_rnnt.txt stdout of the PRE-refactor example on rnnt/ + audio.wav + +The goldens are NOT written by this script: they were captured by running the +pre-refactor `parakeet-transcribe` binary (main @ f98e1e48) on these exact +fixtures, and tests/vllm/models/test_parakeet_transcription_fold.cpp gates the +post-refactor library path byte-identical against them. Regenerating the +CHECKPOINTS is safe (every byte is a pure function of the seeds below); +regenerating the GOLDENS requires rebuilding the pre-refactor example. + +Weight filler (shared contract with nothing β€” the committed bytes are the +fixture): w[i] = (((seed*31 + i) % 17) - 8) / 16, exactly representable in f32. +BatchNorm running_var uses a strictly positive variant. The tokenizer is the +published Parakeet shape: BPE model, `Metaspace` pre_tokenizer with +`split: true` + `prepend_scheme: "always"`, and a `Metaspace` DECODER β€” the +exact file `vllm::Tokenizer` used to refuse (the reason the example carried a +private DecodeIds). +""" + +from __future__ import annotations + +import json +import math +import struct +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +OUT = ROOT / "tests/vllm/models/fixtures/parakeet_e2e" + +# ── tiny geometry (shared by both heads) ───────────────────────────────────── +HID = 16 # encoder hidden_size +LAYERS = 1 # encoder layers +HEADS = 2 # attention heads (head_dim 8) +INTER = 32 # feed-forward intermediate +MELS = 8 # num_mel_bins +SUB_C = 4 # subsampling_conv_channels +SUB_F = 4 # subsampling_factor (2 conv stages, stride 2) +KCONV = 3 # conformer depthwise kernel +KSUB = 3 # subsampling conv kernel +VOCAB = 8 # vocab incl. blank +BLANK = 7 # == pad_token_id (CTC blank) == blank_token_id (RNN-T) +DEC_D = 8 # transducer decoder_hidden_size +DEC_L = 1 # transducer LSTM layers + + +def fill(n: int, seed: int) -> bytes: + """f32 little-endian: (((seed*31 + i) % 17) - 8) / 16 β€” exact in float32.""" + return b"".join( + struct.pack(" bytes: + """Strictly positive variant for BatchNorm running_var: (((...)%17)+9)/16.""" + return b"".join( + struct.pack(" None: + header: dict[str, dict] = {} + data = b"" + for name, shape, raw in tensors: + start = len(data) + data += raw + header[name] = { + "dtype": "F32", + "shape": shape, + "data_offsets": [start, len(data)], + } + blob = json.dumps(header, sort_keys=True).encode() + path.write_bytes(struct.pack(" list[tuple[str, list[int], bytes]]: + s = seed0 + t: list[tuple[str, list[int], bytes]] = [] + + def add(name: str, shape: list[int], pos: bool = False) -> None: + nonlocal s + n = math.prod(shape) + t.append((name, shape, (fill_pos if pos else fill)(n, s))) + s += 1 + + add("encoder.subsampling.layers.0.weight", [SUB_C, 1, KSUB, KSUB]) + add("encoder.subsampling.layers.0.bias", [SUB_C]) + add("encoder.subsampling.layers.2.weight", [SUB_C, 1, KSUB, KSUB]) + add("encoder.subsampling.layers.2.bias", [SUB_C]) + add("encoder.subsampling.layers.3.weight", [SUB_C, SUB_C, 1, 1]) + add("encoder.subsampling.layers.3.bias", [SUB_C]) + out_freq = MELS // (2 * 2) + add("encoder.subsampling.linear.weight", [HID, SUB_C * out_freq]) + add("encoder.subsampling.linear.bias", [HID]) + for layer in range(LAYERS): + base = f"encoder.layers.{layer}." + for ff in ("feed_forward1.", "feed_forward2."): + add(base + ff + "linear1.weight", [INTER, HID]) + add(base + ff + "linear1.bias", [INTER]) + add(base + ff + "linear2.weight", [HID, INTER]) + add(base + ff + "linear2.bias", [HID]) + for proj in ("q_proj", "k_proj", "v_proj", "o_proj"): + add(base + f"self_attn.{proj}.weight", [HID, HID]) + add(base + f"self_attn.{proj}.bias", [HID]) + add(base + "self_attn.relative_k_proj.weight", [HID, HID]) + add(base + "self_attn.bias_u", [HEADS, HID // HEADS]) + add(base + "self_attn.bias_v", [HEADS, HID // HEADS]) + add(base + "conv.pointwise_conv1.weight", [2 * HID, HID, 1]) + add(base + "conv.pointwise_conv1.bias", [2 * HID]) + add(base + "conv.depthwise_conv.weight", [HID, 1, KCONV]) + add(base + "conv.depthwise_conv.bias", [HID]) + add(base + "conv.norm.weight", [HID]) + add(base + "conv.norm.bias", [HID]) + add(base + "conv.norm.running_mean", [HID]) + add(base + "conv.norm.running_var", [HID], pos=True) + add(base + "conv.pointwise_conv2.weight", [HID, HID, 1]) + add(base + "conv.pointwise_conv2.bias", [HID]) + for norm in ( + "norm_feed_forward1", + "norm_self_att", + "norm_conv", + "norm_feed_forward2", + "norm_out", + ): + add(base + norm + ".weight", [HID]) + add(base + norm + ".bias", [HID]) + return t + + +def encoder_config() -> dict: + return { + "hidden_size": HID, + "num_hidden_layers": LAYERS, + "num_attention_heads": HEADS, + "intermediate_size": INTER, + "num_mel_bins": MELS, + "subsampling_conv_channels": SUB_C, + "subsampling_factor": SUB_F, + "subsampling_conv_kernel_size": KSUB, + "subsampling_conv_stride": 2, + "conv_kernel_size": KCONV, + "attention_bias": True, + "convolution_bias": True, + "scale_input": True, + "hidden_act": "silu", + } + + +def tokenizer_json() -> dict: + # The published Parakeet tokenizer shape: BPE vocab + Metaspace(split=true) + # pre_tokenizer + Metaspace DECODER (`{"type": "Metaspace", "replacement": + # "▁", "prepend_scheme": "always", "split": true}` on every published + # checkpoint β€” see examples/parakeet_transcribe pre-refactor DecodeIds). + vocab = { + "▁the": 0, + "▁cat": 1, + "▁sat": 2, + "at": 3, + "he": 4, + "s": 5, + "▁on": 6, + } + return { + "version": "1.0", + "pre_tokenizer": { + "type": "Metaspace", + "replacement": "▁", + "prepend_scheme": "always", + "split": True, + }, + "decoder": { + "type": "Metaspace", + "replacement": "▁", + "prepend_scheme": "always", + "split": True, + }, + "model": { + "type": "BPE", + "unk_token": None, + "vocab": vocab, + "merges": [], + }, + "added_tokens": [ + {"id": BLANK, "content": "", "special": True}, + ], + } + + +def write_wav(path: Path) -> None: + # Four strongly different 62.5 ms segments (tone / loud high tone / chirp / + # deterministic "noise"), so the per-frame features β€” and therefore the + # greedy ids β€” VARY across the clip. A stationary tone made every frame + # argmax to the same id, which was a degenerate golden: it never exercised + # the Metaspace first-piece vs later-piece decode distinction. + n = 4000 # 0.25 s @ 16 kHz + samples = bytearray() + for i in range(n): + seg = i // 1000 + if seg == 0: + v = 0.45 * math.sin(2 * math.pi * 440.0 * i / 16000.0) + elif seg == 1: + v = 0.9 * math.sin(2 * math.pi * 3100.0 * i / 16000.0) + elif seg == 2: + f = 200.0 + (i - 2000) * 3.0 # 200 -> 3200 Hz chirp + v = 0.6 * math.sin(2 * math.pi * f * i / 16000.0) + else: + v = ((((i * 2654435761) >> 7) & 0xFFFF) / 32768.0 - 1.0) * 0.5 + samples += struct.pack(" None: + dir_.mkdir(parents=True, exist_ok=True) + cfg = { + "architectures": ["ParakeetForCTC"], + "model_type": "parakeet_ctc", + "vocab_size": VOCAB, + "pad_token_id": BLANK, + "encoder_config": encoder_config(), + } + (dir_ / "config.json").write_text(json.dumps(cfg, indent=1) + "\n") + (dir_ / "tokenizer.json").write_text( + json.dumps(tokenizer_json(), ensure_ascii=False, indent=1) + "\n" + ) + # Head seeds chosen (probed over the fixed encoder + clip) so the greedy + # ids VARY across frames: 907/500 yields ids `3 4 3` -> "atheat", a + # three-piece transcript rather than a degenerate single token. + tensors = encoder_tensors(seed0=1) + tensors.append(("ctc_head.weight", [VOCAB, HID, 1], fill(VOCAB * HID, 907))) + tensors.append(("ctc_head.bias", [VOCAB], fill(VOCAB, 500))) + write_safetensors(dir_ / "model.safetensors", tensors) + + +def emit_rnnt(dir_: Path) -> None: + dir_.mkdir(parents=True, exist_ok=True) + cfg = { + "architectures": ["ParakeetForRNNT"], + "model_type": "parakeet_rnnt", + "vocab_size": VOCAB, + "blank_token_id": BLANK, + "pad_token_id": 2, + "decoder_hidden_size": DEC_D, + "num_decoder_layers": DEC_L, + "max_symbols_per_step": 3, + "hidden_act": "relu", + "encoder_config": encoder_config(), + } + (dir_ / "config.json").write_text(json.dumps(cfg, indent=1) + "\n") + (dir_ / "generation_config.json").write_text( + json.dumps({"decoder_start_token_id": BLANK}, indent=1) + "\n" + ) + (dir_ / "tokenizer.json").write_text( + json.dumps(tokenizer_json(), ensure_ascii=False, indent=1) + "\n" + ) + tensors = encoder_tensors(seed0=101) + tensors.append( + ("encoder_projector.weight", [DEC_D, HID], fill(DEC_D * HID, 801)) + ) + tensors.append(("encoder_projector.bias", [DEC_D], fill(DEC_D, 802))) + tensors.append( + ("decoder.embedding.weight", [VOCAB, DEC_D], fill(VOCAB * DEC_D, 803)) + ) + for layer in range(DEC_L): + s = 810 + 4 * layer + tensors.append( + (f"decoder.lstm.weight_ih_l{layer}", [4 * DEC_D, DEC_D], + fill(4 * DEC_D * DEC_D, s)) + ) + tensors.append( + (f"decoder.lstm.weight_hh_l{layer}", [4 * DEC_D, DEC_D], + fill(4 * DEC_D * DEC_D, s + 1)) + ) + tensors.append( + (f"decoder.lstm.bias_ih_l{layer}", [4 * DEC_D], fill(4 * DEC_D, s + 2)) + ) + tensors.append( + (f"decoder.lstm.bias_hh_l{layer}", [4 * DEC_D], fill(4 * DEC_D, s + 3)) + ) + tensors.append( + ("decoder.decoder_projector.weight", [DEC_D, DEC_D], + fill(DEC_D * DEC_D, 830)) + ) + tensors.append(("decoder.decoder_projector.bias", [DEC_D], fill(DEC_D, 831))) + tensors.append(("joint.head.weight", [VOCAB, DEC_D], fill(VOCAB * DEC_D, 840))) + tensors.append(("joint.head.bias", [VOCAB], fill(VOCAB, 841))) + write_safetensors(dir_ / "model.safetensors", tensors) + + +def main() -> int: + OUT.mkdir(parents=True, exist_ok=True) + write_wav(OUT / "audio.wav") + emit_ctc(OUT / "ctc") + emit_rnnt(OUT / "rnnt") + print(f"wrote {OUT}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/capi/vllm_c.cpp b/src/capi/vllm_c.cpp index 0d9733fad..600e8d550 100644 --- a/src/capi/vllm_c.cpp +++ b/src/capi/vllm_c.cpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -39,14 +40,25 @@ #include "vllm/entrypoints/openai/tool_parsers/detect.h" // DetectToolParser #include "vllm/entrypoints/openai/reasoning_parsers/abstract.h" // get_reasoning_parser #include "vllm/entrypoints/openai/reasoning_parsers/detect.h" // DetectReasoningParser +#include "vllm/model_executor/models/model_registry.h" // refuse-by-task (v11) +#include "vllm/multimodal/parakeet_transcription.h" // vllm_transcribe (v11) #include "vllm/outputs.h" #include "vllm/sampling_params.h" +#include "vllm/transformers_utils/hf_config.h" // PeekHfArchitectures (v11) #include "vllm/version.h" #include "vllm/v1/engine/async_llm.h" -// The opaque handle: owns the whole C++ engine stack behind LoadedEngine. +// The opaque handle: owns the whole C++ engine stack behind LoadedEngine β€” +// OR, since ABI v11, a transcription stack (ParakeetTranscriber) when the +// model directory resolves to a SupportsTranscription-ONLY architecture. +// Exactly one of `loaded` / `transcriber` is set; every text entry point +// guards on `loaded` (RequireTextEngine) and vllm_transcribe on `transcriber`, +// so the two task families refuse each other cleanly instead of crashing. struct vllm_engine { std::unique_ptr loaded; + // ABI v11 transcription stack (the ONE library seam the server route and the + // parakeet-transcribe example also drive). Null for text engines. + std::unique_ptr transcriber; // Monotonic per-handle request-id source. Each vllm_complete[_stream] call // uses a FRESH id so a request left in-flight by a mid-call exception can never // collide with a later call's id β€” a collision would make LLMEngine.add_request @@ -128,6 +140,19 @@ thread_local std::string g_last_error; void SetError(const std::string& msg) { g_last_error = msg; } void ClearError() { g_last_error.clear(); } +// ABI v11 refuse-by-task: true when the handle owns the TEXT engine stack. +// A transcription-only handle (Parakeet) reports an actionable error instead +// of dereferencing the null LoadedEngine β€” the SupportsTranscription-only +// mirror of vLLM excluding "generate" from supported_tasks +// (vllm/model_executor/models/interfaces.py:1118). +bool RequireTextEngine(const vllm_engine* engine, const char* fn) { + if (engine->loaded != nullptr) return true; + SetError(std::string(fn) + + ": this engine was loaded from a transcription-only checkpoint " + "(Parakeet); it has no text-generation path β€” use vllm_transcribe"); + return false; +} + // Heap-copy a std::string into a caller-owned NUL-terminated C string (freed via // vllm_string_free / vllm_completion_free). Returns nullptr on allocation // failure. @@ -575,6 +600,36 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, return VLLM_ERR_INVALID_ARGUMENT; } + // ABI v11 task dispatch: a directory whose config.json architectures + // resolve to a SupportsTranscription-ONLY registration (Parakeet + // CTC/RNNT/TDT) gets the TRANSCRIPTION stack β€” the same library seam the + // server's /v1/audio/transcriptions route and the parakeet-transcribe + // example drive. The peek is non-throwing and narrow: every other path, + // including unknown architectures and .gguf files, is byte-identical to + // pre-v11 (FromModelDir owns the diagnosis). + if (const std::vector archs = + vllm::PeekHfArchitectures(std::string(params->model_path) + + "/config.json"); + !archs.empty()) { + const vllm::ModelRegistration* peek = nullptr; + try { + peek = &vllm::ModelRegistry::Resolve( + std::span(archs)); + } catch (const std::exception&) { + peek = nullptr; + } + if (peek != nullptr && peek->info.supports_transcription_only) { + auto* handle = new vllm_engine; + handle->transcriber = + std::make_unique( + vllm::multimodal::ParakeetTranscriber::FromDir( + params->model_path)); + handle->model_path = params->model_path; + *out = handle; + ClearError(); + return VLLM_OK; + } + } auto loaded = vllm::entrypoints::LoadedEngine::FromModelDir(params->model_path, ep); auto* handle = new vllm_engine; @@ -619,6 +674,7 @@ VLLM_API vllm_status vllm_complete(vllm_engine* engine, const char* prompt, SetError("vllm_complete: engine, prompt or params is null"); return VLLM_ERR_INVALID_ARGUMENT; } + if (!RequireTextEngine(engine, "vllm_complete")) return VLLM_ERR_INVALID_ARGUMENT; try { const vllm::SamplingParams sp = ToSamplingParams(*params, vllm::RequestOutputKind::kCumulative); @@ -674,6 +730,7 @@ VLLM_API vllm_status vllm_complete_stream(vllm_engine* engine, SetError("vllm_complete_stream: engine, prompt, params or cb is null"); return VLLM_ERR_INVALID_ARGUMENT; } + if (!RequireTextEngine(engine, "vllm_complete_stream")) return VLLM_ERR_INVALID_ARGUMENT; try { // DELTA output_kind: each step yields one incremental delta (mirrors the M3.1 // OpenAI streaming path, serving_completion.cpp). PostInit runs inside. @@ -742,6 +799,7 @@ VLLM_API vllm_status vllm_request_submit( SetError("vllm_request_submit: engine, prompt, params or cb is null"); return VLLM_ERR_INVALID_ARGUMENT; } + if (!RequireTextEngine(engine, "vllm_request_submit")) return VLLM_ERR_INVALID_ARGUMENT; try { vllm::SamplingParams sp = ToSamplingParams(*params, vllm::RequestOutputKind::kDelta); @@ -863,6 +921,7 @@ VLLM_API vllm_status vllm_chat(vllm_engine* engine, const char* request_json, SetError("vllm_chat: engine or request_json is null"); return VLLM_ERR_INVALID_ARGUMENT; } + if (!RequireTextEngine(engine, "vllm_chat")) return VLLM_ERR_INVALID_ARGUMENT; try { vllm::entrypoints::openai::ChatCompletionRequest request = ParseChatRequest(request_json); @@ -902,6 +961,7 @@ VLLM_API vllm_status vllm_chat_stream(vllm_engine* engine, SetError("vllm_chat_stream: engine, request_json or cb is null"); return VLLM_ERR_INVALID_ARGUMENT; } + if (!RequireTextEngine(engine, "vllm_chat_stream")) return VLLM_ERR_INVALID_ARGUMENT; try { vllm::entrypoints::openai::ChatCompletionRequest request = ParseChatRequest(request_json); @@ -953,6 +1013,105 @@ VLLM_API vllm_status vllm_chat_stream(vllm_engine* engine, } } +// ── Audio transcription (ABI v11) ─────────────────────────────────────────── + +VLLM_API vllm_transcription_params vllm_transcription_params_default(void) { + vllm_transcription_params p; + p.audio_path = nullptr; + p.pcm = nullptr; + p.n_samples = 0; + p.sample_rate = 0; + return p; +} + +VLLM_API vllm_status vllm_transcribe(vllm_engine* engine, + const vllm_transcription_params* params, + vllm_transcription* out) { + if (out == nullptr) { + SetError("vllm_transcribe: out is null"); + return VLLM_ERR_INVALID_ARGUMENT; + } + out->text = nullptr; + out->token_ids = nullptr; + out->n_token_ids = 0; + out->has_text = 0; + if (engine == nullptr || params == nullptr) { + SetError("vllm_transcribe: engine or params is null"); + return VLLM_ERR_INVALID_ARGUMENT; + } + if (engine->transcriber == nullptr) { + SetError( + "vllm_transcribe: this engine is a text-generation engine (no " + "SupportsTranscription architecture); use the completion/chat entry " + "points, or load a Parakeet checkpoint for transcription"); + return VLLM_ERR_INVALID_ARGUMENT; + } + const bool has_path = params->audio_path != nullptr; + const bool has_pcm = params->pcm != nullptr; + if (has_path == has_pcm) { + SetError( + "vllm_transcribe: set exactly ONE input β€” audio_path (a 16-bit PCM " + "mono WAV) or pcm+n_samples+sample_rate"); + return VLLM_ERR_INVALID_ARGUMENT; + } + if (has_pcm && (params->n_samples <= 0 || params->sample_rate <= 0)) { + SetError("vllm_transcribe: pcm requires n_samples > 0 and sample_rate > 0"); + return VLLM_ERR_INVALID_ARGUMENT; + } + try { + const vllm::multimodal::ParakeetTranscription result = + has_path ? engine->transcriber->TranscribeWavFile(params->audio_path) + : engine->transcriber->Transcribe(params->pcm, + params->n_samples, + params->sample_rate); + int32_t* ids = nullptr; + if (!result.token_ids.empty()) { + ids = static_cast( + std::malloc(result.token_ids.size() * sizeof(int32_t))); + if (ids == nullptr) { + SetError("vllm_transcribe: out-of-memory copying token ids"); + return VLLM_ERR_RUNTIME; + } + std::memcpy(ids, result.token_ids.data(), + result.token_ids.size() * sizeof(int32_t)); + } + if (result.has_text) { + char* text = DupString(result.text); + if (text == nullptr) { + std::free(ids); + SetError("vllm_transcribe: out-of-memory copying transcript"); + return VLLM_ERR_RUNTIME; + } + out->text = text; + out->has_text = 1; + } + out->token_ids = ids; + out->n_token_ids = static_cast(result.token_ids.size()); + ClearError(); + return VLLM_OK; + } catch (const std::exception& e) { + // One catch for the whole pipeline (unreadable/undecodable audio, wrong + // sample rate, forward failure): VLLM_ERR_RUNTIME with the cause named, + // matching vllm_complete's convention. The cheap argument-shape errors + // were already reported as VLLM_ERR_INVALID_ARGUMENT above. + SetError(std::string("vllm_transcribe: ") + e.what()); + return VLLM_ERR_RUNTIME; + } catch (...) { + SetError("vllm_transcribe: unknown error"); + return VLLM_ERR_UNKNOWN; + } +} + +VLLM_API void vllm_transcription_free(vllm_transcription* out) { + if (out == nullptr) return; + std::free(out->text); + std::free(out->token_ids); + out->text = nullptr; + out->token_ids = nullptr; + out->n_token_ids = 0; + out->has_text = 0; +} + VLLM_API void vllm_string_free(char* s) { std::free(s); } VLLM_API void vllm_completion_free(vllm_completion* out) { diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 78be97bbe..64b08070b 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -891,6 +891,37 @@ std::unique_ptr LoadedEngine::FromModelDir( const std::string config_path = (dir / "config.json").string(); const std::string tokenizer_path = (dir / "tokenizer.json").string(); + // Refuse-by-task (ARCH-ONE-SURFACE ROW 1), BEFORE the full HfConfig parse: a + // SupportsTranscription-ONLY architecture (Parakeet CTC/RNNT/TDT) has no + // text-generation path, so the text engine must not be built around it β€” + // mirror of vLLM excluding "generate" from supported_tasks for + // supports_transcription_only models (interfaces.py:1118). The peek is + // deliberately narrow: only a config whose architectures RESOLVE to a + // transcription-only registration takes this exit (its config shape β€” e.g. + // hidden_size nested under encoder_config β€” would otherwise fail the text + // HfConfig parse below with a misleading message); every other model, known + // or unknown, falls through with error ordering unchanged. The C ABI routes + // such a directory to the transcription stack before reaching here + // (vllm_c.cpp), so this fires only for a text-only consumer (server --task + // generate, vllm-cli, bench). + if (const std::vector archs = + vllm::PeekHfArchitectures(config_path); + !archs.empty()) { + const ModelRegistration* peek = nullptr; + try { + peek = &ModelRegistry::Resolve(std::span(archs)); + } catch (const std::exception&) { + peek = nullptr; // unknown arch: the existing path owns the diagnosis + } + if (peek != nullptr && peek->info.supports_transcription_only) { + throw std::runtime_error( + "Model architecture " + std::string(peek->architecture) + + " supports transcription only (no text generation). Use " + "vllm_transcribe on the C ABI or the server's " + "/v1/audio/transcriptions instead of the text-generation entry " + "points."); + } + } HfConfig config = vllm::LoadHfConfig(config_path); const ModelRegistration& registration = ModelRegistry::Resolve(config); tok::Tokenizer tokenizer = tok::Tokenizer::FromHfJson(tokenizer_path); diff --git a/src/vllm/entrypoints/openai/api_server.cpp b/src/vllm/entrypoints/openai/api_server.cpp index a3d31c20c..624f46b3d 100644 --- a/src/vllm/entrypoints/openai/api_server.cpp +++ b/src/vllm/entrypoints/openai/api_server.cpp @@ -108,12 +108,23 @@ ApiServer::ApiServer(OpenAIServingCompletion& completion, OpenAIServingChat& chat, OpenAIServingModels& models, std::string version, size_t max_concurrent_streams, HttpWorkerPoolMode worker_pool_mode) - : completion_(completion), - chat_(chat), + : completion_(&completion), + chat_(&chat), models_(models), version_(std::move(version)), impl_(std::make_unique(max_concurrent_streams, worker_pool_mode)) {} +// Serving-less construction (transcription-only servers, ARCH-ONE-SURFACE +// ROW 1): no AsyncLLM exists, so the generate handlers stay null and their +// routes are not registered β€” vLLM's task-conditional route registration +// (api_server.py:255-265) expressed at construction. +ApiServer::ApiServer(OpenAIServingModels& models, std::string version, + size_t max_concurrent_streams, + HttpWorkerPoolMode worker_pool_mode) + : models_(models), + version_(std::move(version)), + impl_(std::make_unique(max_concurrent_streams, worker_pool_mode)) {} + ApiServer::~ApiServer() { // Drain the async /v1/videos workers before the job store they write into is // destroyed. Threads are joined, never detached, precisely so this ordering is @@ -130,6 +141,14 @@ ApiServer::~ApiServer() { ApiServer::DispatchResult ApiServer::handle_completions( const std::string& request_body) { + if (completion_ == nullptr) { + // vLLM's api_router `if handler is None: raise NotImplementedError` mirror + // for a serving-less (transcription-only) server; the socket layer never + // registers the route in that mode, so this answers direct dispatch only. + return MakeError(500, "InternalServerError", + "The model does not support Completions API " + "(transcription-only server)"); + } // completion/api_router.py:46 (create_completion): parse β†’ check_model β†’ // handler β†’ JSON (non-stream) or text/event-stream (stream). nlohmann::json body; @@ -156,8 +175,8 @@ ApiServer::DispatchResult ApiServer::handle_completions( try { std::unique_lock legacy_lock(impl_->legacy_engine_mutex, std::defer_lock); - if (!completion_.uses_async_engine()) legacy_lock.lock(); - result = completion_.create_completion(request); + if (!completion_->uses_async_engine()) legacy_lock.lock(); + result = completion_->create_completion(request); } catch (const std::exception& e) { // DISCRIMINATOR: attribute a 500 to its endpoint + model + raw cause so a // benchmark driver that only sees the generic HTTP body can still recover @@ -183,6 +202,11 @@ ApiServer::DispatchResult ApiServer::handle_completions( ApiServer::DispatchResult ApiServer::handle_chat_completions( const std::string& request_body) { + if (chat_ == nullptr) { + return MakeError(500, "InternalServerError", + "The model does not support Chat Completions API " + "(transcription-only server)"); + } // chat_completion/api_router.py:53 (create_chat_completion). nlohmann::json body; try { @@ -208,8 +232,8 @@ ApiServer::DispatchResult ApiServer::handle_chat_completions( try { std::unique_lock legacy_lock(impl_->legacy_engine_mutex, std::defer_lock); - if (!chat_.uses_async_engine()) legacy_lock.lock(); - result = chat_.create_chat_completion(request); + if (!chat_->uses_async_engine()) legacy_lock.lock(); + result = chat_->create_chat_completion(request); } catch (const std::exception& e) { std::cerr << "api-server: 500 endpoint=/v1/chat/completions model=" << request.model.value_or("") << " what=" << e.what() << "\n"; @@ -288,6 +312,59 @@ std::string ApiServer::video_model_warning( "'); generated with the video model this server was started with"; } +ApiServer::DispatchResult ApiServer::handle_audio_transcriptions( + const std::string& file_bytes, const std::string& response_format) const { + // Mirror of vLLM speech_to_text/transcription: api_router.py:31 + // `create_transcriptions` reads the multipart upload + // (read_upload_with_limit) and hands the bytes to + // OpenAIServingTranscription.create_transcription (serving.py:50), which + // answers TranscriptionResponse {"text": ...} for response_format json and + // the raw text otherwise. The transcription itself runs through the ONE + // library seam (ParakeetTranscriber) β€” the same code path vllm_transcribe + // drives, so HTTP and FFI cannot drift. + if (!transcriber_) { + // The api_router `if handler is None: raise NotImplementedError` mirror; + // the socket layer never registers the route without a transcriber. + return MakeError(500, "InternalServerError", + "The model does not support Transcriptions API"); + } + if (file_bytes.empty()) { + return MakeError(400, "BadRequestError", + "Expected a non-empty `file` upload (16-bit PCM mono " + "RIFF/WAVE)"); + } + const std::string fmt = response_format.empty() ? "json" : response_format; + if (fmt != "json" && fmt != "text") { + // verbose_json / srt / vtt are NAMED RESIDUALS of this fold (protocol.py + // AudioResponseFormat lists them; nothing here produces segment timing). + return MakeError(400, "BadRequestError", + "response_format '" + fmt + + "' is not supported (supported: json, text; " + "verbose_json/srt/vtt are named residuals)"); + } + try { + const ::vllm::multimodal::ParakeetTranscription result = transcriber_( + reinterpret_cast(file_bytes.data()), file_bytes.size()); + if (!result.has_text) { + return MakeError(500, "InternalServerError", + "the checkpoint ships no tokenizer.json, so ids-only " + "transcription has no OpenAI response shape"); + } + DispatchResult r; + if (fmt == "text") { + r.content_type = "text/plain; charset=utf-8"; + r.body = result.text; + } else { + r.body = nlohmann::json{{"text", result.text}}.dump(); + } + return r; + } catch (const std::exception& e) { + // Undecodable audio (not RIFF/WAVE, not PCM16 mono, wrong sample rate) is + // a caller error; the pipeline names the cause. + return MakeError(400, "BadRequestError", e.what()); + } +} + ApiServer::DispatchResult ApiServer::handle_videos( const std::string& request_body) { // vLLM-Omni's ASYNC video endpoint: validate, enqueue, and return the job id @@ -495,7 +572,12 @@ ApiServer::DispatchResult ApiServer::handle_tokenize( const bool render_generation_prompt = add_generation_prompt && !continue_final_message; try { - prompt = chat_.prompt_fn()(messages, render_generation_prompt, tools); + if (chat_ == nullptr) { + return MakeError(500, "InternalServerError", + "tokenize: the chat form needs the chat template of a " + "text-generation server (transcription-only server)"); + } + prompt = chat_->prompt_fn()(messages, render_generation_prompt, tools); } catch (const std::exception& e) { return MakeError(400, "BadRequestError", std::string("Chat template render failed: ") + e.what()); @@ -778,14 +860,22 @@ void ApiServer::register_routes() { } }; - server.Post("/v1/completions", - [this, write](const httplib::Request& req, httplib::Response& res) { - write(handle_completions(req.body), res); - }); - server.Post("/v1/chat/completions", - [this, write](const httplib::Request& req, httplib::Response& res) { - write(handle_chat_completions(req.body), res); - }); + // TASK-CONDITIONAL (mirrors vLLM registering the generate routes only when + // "generate" is in supported_tasks, api_server.py:255-265): a serving-less + // (transcription-only) server has no completion/chat handlers, so the two + // generate routes are NOT registered and answer 404. + if (completion_ != nullptr) { + server.Post("/v1/completions", + [this, write](const httplib::Request& req, httplib::Response& res) { + write(handle_completions(req.body), res); + }); + } + if (chat_ != nullptr) { + server.Post("/v1/chat/completions", + [this, write](const httplib::Request& req, httplib::Response& res) { + write(handle_chat_completions(req.body), res); + }); + } server.Get("/v1/models", [this, write](const httplib::Request&, httplib::Response& res) { write(handle_models(), res); @@ -815,6 +905,29 @@ void ApiServer::register_routes() { write(handle_server_info(), res); }); + if (transcriber_) { + // Parakeet ASR (ARCH-ONE-SURFACE ROW 1). Registered ONLY when a + // transcriber is attached, so a text server answers 404 exactly as before. + // The multipart shape mirrors vLLM's create_transcriptions + // (speech_to_text/transcription/api_router.py:31): the audio arrives as + // the `file` upload, `response_format` as an ordinary form field. + server.Post("/v1/audio/transcriptions", + [this, write](const httplib::Request& req, + httplib::Response& res) { + if (!req.form.has_file("file")) { + write(MakeError(400, "BadRequestError", + "multipart/form-data with a `file` upload " + "is required"), + res); + return; + } + write(handle_audio_transcriptions( + req.form.get_file("file").content, + req.form.get_field("response_format")), + res); + }); + } + if (video_runner_) { // MiniMax-H3. Registered ONLY when a runner is attached, so a server built // without video support answers 404 exactly as before. diff --git a/src/vllm/model_executor/models/parakeet_registry.cpp b/src/vllm/model_executor/models/parakeet_registry.cpp new file mode 100644 index 000000000..6cdcb13b4 --- /dev/null +++ b/src/vllm/model_executor/models/parakeet_registry.cpp @@ -0,0 +1,143 @@ +// Parakeet (`ParakeetForCTC` / `ParakeetForRNNT` / `ParakeetForTDT`) registry +// TU β€” the ADDITIVE self-registration seam for the ARCH-ONE-SURFACE ROW 1 +// audio-transcription fold. Follows the kimi_k3_registry.cpp refuse-stub +// precedent exactly: a NEW translation unit with REGISTER_VLLM_MODEL lines and +// ZERO edit to any shared array. +// +// WHY these registrations exist: so an HF Parakeet checkpoint directory +// RESOLVES from config.json `architectures` and the entrypoints can dispatch +// BY TASK β€” the ModelInfo carries the SupportsTranscription mirror +// (vllm/model_executor/models/interfaces.py:1110-1118, with +// `supports_transcription_only` true: these archs have NO text-generation +// path), which LoadedEngine::FromModelDir turns into a clean refusal pointing +// at vllm_transcribe / /v1/audio/transcriptions. The transcription work itself +// runs through vllm::multimodal::ParakeetTranscriber +// (include/vllm/multimodal/parakeet_transcription.h), NOT through +// ModelRegistry::Forward β€” an encoder+greedy-decode pipeline has no +// logits-per-step contract to satisfy. +// +// BEYOND-PIN BREADTH, recorded: the pinned vLLM registers NO standalone +// Parakeet architecture β€” upstream's Parakeet is the audio-encoder COMPONENT +// of NemotronH_Nano_VL_V2 (vllm/model_executor/models/registry.py:511-513), +// wrapping transformers' encoder (parakeet.py:37,62). The standalone +// CTC/RNNT/TDT classes we resolve here are transformers-`main` classes +// (modeling_parakeet.py: ParakeetForCTC:675, ParakeetForRNNT:922, +// ParakeetForTDT:1052) β€” grounded per file:line in parakeet_encoder.h / +// parakeet_transducer.h and marked beyond-pin in .agents/model-matrix.md. +// +// A registered Parakeet must NOT crash the text-generation paths: every +// factory hook below refuses with the same actionable message instead of +// VT_CHECK-aborting, and the KV spec is a never-exercised placeholder (the +// kimi_k3 stub precedent) so resolution-time plumbing that sizes caches +// cannot trip on a null factory field. +#include "vllm/model_executor/models/model_registry.h" + +#include +#include +#include + +#include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits carrier +#include "vllm/v1/kv_cache_dtype.h" +#include "vllm/v1/kv_cache_interface.h" +#include "vt/dtype.h" + +namespace vllm { +namespace { + +[[noreturn]] void RefuseTextGeneration() { + throw std::runtime_error( + "Parakeet architectures support transcription only " + "(SupportsTranscription, no text-generation path): drive the checkpoint " + "through vllm_transcribe on the C ABI or the server's " + "/v1/audio/transcriptions, not the text-generation entry points"); +} + +// registry.py _ModelInfo mirror: NOT a text-generation model; ASR-capable and +// ASR-ONLY (interfaces.py:1116-1118). +inline constexpr ModelInfo kParakeetInfo{ + .is_text_generation_model = false, + .is_pooling_model = false, + .is_hybrid = false, + .has_inner_state = false, + .supports_multimodal = false, + .supports_transcription = true, + .supports_transcription_only = true, + .score_type = "bi-encoder", +}; + +void ParseParakeetConfig(const HfConfig& config) { + // Nothing to validate at resolution time: the transcription seam's own + // loader (LoadParakeetForCTC / LoadParakeetTransducer) parses config.json + // and fails loudly per missing/misshaped field. + (void)config; +} + +std::unique_ptr LoadParakeetRefused( + const ModelRegistration& registration, const HfConfig& config, + const ModelSource& source) { + (void)registration; + (void)config; + (void)source; + RefuseTextGeneration(); +} + +void PrepareParakeetRefused(LoadedModel& model, const HfConfig& config, + vt::Queue& queue) { + (void)model; + (void)config; + (void)queue; + RefuseTextGeneration(); +} + +ForwardLogits ForwardParakeetRefused(LoadedModel& model, + const ModelForwardInput& input) { + (void)model; + (void)input; + // VT_CHECK(false, ...) β€” the registry's REFUSE-stub contract the + // runner-routing gate recognizes (classify_body REFUSE, the kimi_k3/ + // deepseek_v4 precedent). It THROWS a clean std::runtime_error + // (vt/dtype.h:11), never aborts, so the refusal stays actionable. + VT_CHECK(false, + "Parakeet architectures support transcription only " + "(SupportsTranscription, no text-generation path): drive the " + "checkpoint through vllm_transcribe on the C ABI or the server's " + "/v1/audio/transcriptions, not the text-generation entry points"); + return {}; +} + +v1::KVCacheConfig MakeParakeetKVCache(const HfConfig& config, int block_size, + int num_blocks) { + // PLACEHOLDER, never exercised: the encoder has no KV cache at all (no + // autoregressive attention), and the refuse-by-task gate in + // LoadedEngine::FromModelDir fires before any cache is built. One minimal + // full-attention group so a caller that sizes specs eagerly cannot crash. + (void)config; + v1::KVCacheConfig kv; + kv.num_blocks = num_blocks; + kv.kv_cache_groups.emplace_back( + std::vector{"encoder"}, + std::make_shared(block_size, /*num_kv_heads=*/1, + /*head_size=*/64, + v1::ResolveKvCacheDType())); + return kv; +} + +const ModelFactory kParakeetFactory{ + .parse_config = &ParseParakeetConfig, + .load_weights = &LoadParakeetRefused, + .prepare = &PrepareParakeetRefused, + .forward = &ForwardParakeetRefused, + .make_kv_cache = &MakeParakeetKVCache, + .is_dense_model = false, +}; + +} // namespace + +REGISTER_VLLM_MODEL(parakeet_ctc, "ParakeetForCTC", kParakeetFactory, + kParakeetInfo) +REGISTER_VLLM_MODEL(parakeet_rnnt, "ParakeetForRNNT", kParakeetFactory, + kParakeetInfo) +REGISTER_VLLM_MODEL(parakeet_tdt, "ParakeetForTDT", kParakeetFactory, + kParakeetInfo) + +} // namespace vllm diff --git a/src/vllm/multimodal/parakeet_transcription.cpp b/src/vllm/multimodal/parakeet_transcription.cpp new file mode 100644 index 000000000..5bebfad35 --- /dev/null +++ b/src/vllm/multimodal/parakeet_transcription.cpp @@ -0,0 +1,141 @@ +// Parakeet ASR transcription seam β€” see include/vllm/multimodal/ +// parakeet_transcription.h for the fold contract and upstream mirror shape. +// +// This is the pipeline `examples/parakeet_transcribe/main.cpp` owned privately +// before the ARCH-ONE-SURFACE ROW 1 fold (main.cpp:176-264 @ f98e1e48), moved +// into the library verbatim in ORDER and SEMANTICS so the fold gate +// (tests/vllm/models/test_parakeet_transcription_fold.cpp) can hold it +// byte-identical to the pre-refactor transcript goldens: +// WAV -> f32 mono DecodeWavPcm16Mono (was: ReadWav16BitMono) +// config.json model_type LoadParakeetModelType (dispatch was main.cpp:210-238) +// log-mel ParakeetAudioProcessor, feature_size = num_mel_bins +// encoder + head ParakeetForCTCForward / ParakeetForTransducerForward +// ids -> text vllm::tok::Tokenizer (was: LoadVocab + DecodeIds β€” +// the Metaspace split=true decoder now lives in the +// tokenizer, so the private copy is deleted) +#include "vllm/multimodal/parakeet_transcription.h" + +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/parakeet_encoder.h" +#include "vllm/model_executor/models/parakeet_transducer.h" +#include "vllm/multimodal/audio_processor.h" +#include "vllm/multimodal/parakeet_audio_processor.h" +#include "vllm/tokenizer/tokenizer.h" +#include "vt/backend.h" + +namespace vllm::multimodal { + +struct ParakeetTranscriber::Impl { + std::string model_type; + ParakeetEncoderConfig enc_cfg; + // Exactly one head is engaged, per config.json `model_type`. + std::optional ctc; + std::optional transducer; + ParakeetTransducerConfig transducer_cfg; + std::optional tokenizer; +}; + +ParakeetTranscriber::ParakeetTranscriber() = default; +ParakeetTranscriber::ParakeetTranscriber(ParakeetTranscriber&&) noexcept = default; +ParakeetTranscriber& ParakeetTranscriber::operator=(ParakeetTranscriber&&) noexcept = + default; +ParakeetTranscriber::~ParakeetTranscriber() = default; + +ParakeetTranscriber ParakeetTranscriber::FromDir(const std::string& dir) { + ParakeetTranscriber t; + t.impl_ = std::make_unique(); + Impl& impl = *t.impl_; + + impl.model_type = LoadParakeetModelType(dir); + if (impl.model_type == "parakeet_rnnt" || impl.model_type == "parakeet_tdt") { + impl.transducer = + LoadParakeetTransducer(dir, &impl.enc_cfg, &impl.transducer_cfg); + } else if (impl.model_type == "parakeet_ctc" || impl.model_type.empty()) { + // An empty model_type falls through to CTC exactly as the pre-refactor + // example did (main.cpp:231-238: only a NON-empty, non-parakeet_ctc value + // was refused). + impl.ctc = LoadParakeetForCTC(dir, &impl.enc_cfg); + } else { + throw std::runtime_error("parakeet: unsupported model_type '" + + impl.model_type + "' (expected parakeet_ctc, " + "parakeet_rnnt or parakeet_tdt)"); + } + + // tokenizer.json is optional (ids-only checkpoints stay usable); a PRESENT + // but unloadable one fails loudly rather than degrading to ids silently. + const std::string tok_path = dir + "/tokenizer.json"; + if (std::filesystem::exists(tok_path)) { + impl.tokenizer.emplace(tok::Tokenizer::FromHfJson(tok_path)); + } + return t; +} + +const std::string& ParakeetTranscriber::model_type() const { + return impl_->model_type; +} + +bool ParakeetTranscriber::has_tokenizer() const { + return impl_->tokenizer.has_value(); +} + +ParakeetTranscription ParakeetTranscriber::Transcribe(const float* samples, + int64_t num_samples, + int sample_rate) const { + const Impl& impl = *impl_; + + // The extractor is driven by the checkpoint's own num_mel_bins (80 on the + // CTC/RNN-T checkpoints, 128 on parakeet-tdt-0.6b-v3), exactly as the + // pre-refactor example configured it (main.cpp:196-202). + ParakeetExtractorConfig ecfg; + ecfg.feature_size = static_cast(impl.enc_cfg.num_mel_bins); + const ParakeetAudioProcessor proc(ecfg); + const ParakeetAudioFeatures feats = + proc.ProcessWaveform(samples, num_samples, sample_rate); + + vt::Backend& cpu = vt::GetBackend(vt::DeviceType::kCPU); + + ParakeetTranscription out; + if (impl.transducer.has_value()) { + const ParakeetTransducerOutput head = ParakeetForTransducerForward( + feats.input_features, feats.num_frames, feats.valid_frames, + *impl.transducer, impl.enc_cfg, impl.transducer_cfg, cpu); + out.token_ids = head.token_ids; + } else { + const ParakeetCTCOutput head = + ParakeetForCTCForward(feats.input_features, feats.num_frames, + feats.valid_frames, *impl.ctc, impl.enc_cfg, cpu); + out.token_ids = head.token_ids; + } + if (impl.tokenizer.has_value()) { + out.text = impl.tokenizer->Decode(out.token_ids); + out.has_text = true; + } + return out; +} + +ParakeetTranscription ParakeetTranscriber::TranscribeWavBytes( + const uint8_t* wav_bytes, size_t num_bytes) const { + const DecodedAudio audio = DecodeWavPcm16Mono(wav_bytes, num_bytes); + return Transcribe(audio.samples.data(), + static_cast(audio.samples.size()), + audio.sampling_rate); +} + +ParakeetTranscription ParakeetTranscriber::TranscribeWavFile( + const std::string& wav_path) const { + std::ifstream f(wav_path, std::ios::binary); + if (!f.good()) { + throw std::runtime_error("parakeet: cannot open " + wav_path); + } + const std::string bytes((std::istreambuf_iterator(f)), + std::istreambuf_iterator()); + return TranscribeWavBytes(reinterpret_cast(bytes.data()), + bytes.size()); +} + +} // namespace vllm::multimodal diff --git a/src/vllm/tokenizer/tokenizer.cpp b/src/vllm/tokenizer/tokenizer.cpp index b58231858..ddf569ccb 100644 --- a/src/vllm/tokenizer/tokenizer.cpp +++ b/src/vllm/tokenizer/tokenizer.cpp @@ -547,15 +547,40 @@ Tokenizer Tokenizer::FromHfJson(const std::string& tokenizer_json_path) { } else { Fail("unsupported Metaspace prepend_scheme \"" + ms_scheme + "\""); } - // split=true would pre-split the metaspace string into per-▁ pretokens - // (MergedWithNext). No checkpoint in scope uses it (Mistral/Gemma set - // split=false); accept it only when we have a golden. Fail loudly rather - // than tokenize a split=true model subtly wrong. - if (ms_split) Fail("Metaspace split=true unsupported (no golden in scope)"); + // split=true pre-splits the metaspace string into per-▁ pretokens + // (SplitDelimiterBehavior::MergedWithNext) before BPE, so merges cannot + // cross a ▁ boundary β€” see EncodePlainSp. This used to Fail ("no golden in + // scope"); the golden arrived with the Parakeet checkpoints (every one + // ships `split: true`), gated by tests/vllm/test_tokenizer_metaspace_split + // .cpp against HF tokenizers 0.22 pre_tokenizers/metaspace.rs semantics + // and the pre-refactor examples/parakeet_transcribe DecodeIds reference. } else { tok.pattern_ = DetectPattern(doc); } + // DECODER selection for the SentencePiece family. Mistral/Gemma ship a + // Sequence decoder (Replace -> ByteFallback -> Fuse -> Strip), which is the + // long-standing default SpDecodeTokens applies. Parakeet ships a bare + // `Metaspace` DECODER node whose decode_chain rule differs observably (the + // first token DROPS every replacement instead of Strip's one leading space), + // so recognize it and record its own replacement + prepend_scheme (HF's + // decode_chain reads the DECODER node's fields, tokenizers 0.22 + // decoders/mod.rs + pre_tokenizers/metaspace.rs). + if (tok.family_ == Family::kSentencePiece) { + const auto dec = doc.find("decoder"); + if (dec != doc.end() && dec->is_object() && + dec->value("type", "") == "Metaspace") { + tok.sp_decoder_metaspace_ = true; + tok.sp_decoder_replacement_ = + dec->value("replacement", std::string("\xE2\x96\x81")); + if (tok.sp_decoder_replacement_.empty()) { + Fail("Metaspace decoder has empty replacement"); + } + tok.sp_decoder_prepend_never_ = + dec->value("prepend_scheme", std::string("always")) == "never"; + } + } + const auto model_it = doc.find("model"); if (model_it == doc.end() || !model_it->is_object()) Fail("missing model"); const json& model = *model_it; @@ -858,68 +883,93 @@ void Tokenizer::EncodePlainSp(std::string_view text, bool at_input_start, } if (s.empty()) return; - // 2) Build the initial BPE symbols. HF constructs the Word BEFORE merging: - // each character maps to itself when present in the vocab, else (with - // byte_fallback) decomposes into its UTF-8 bytes as "<0xNN>" tokens, else - // becomes unk. Merges then run over these symbols. + // 2+3 per PRETOKEN) Build the initial BPE symbols, merge, map to ids. HF + // constructs the Word BEFORE merging: each character maps to itself when + // present in the vocab, else (with byte_fallback) decomposes into its + // UTF-8 bytes as "<0xNN>" tokens, else becomes unk. Merges then run over + // these symbols; fuse_unk collapses consecutive unks WITHIN the pretoken. static const std::string kUnk("\x01\x01unk\x01\x01"); // never a real symbol - std::vector symbols; - size_t pos = 0; - while (pos < s.size()) { - const size_t begin = pos; - (void)DecodeUtf8(s, pos); - std::string ch = s.substr(begin, pos - begin); - if (vocab_.find(ch) != vocab_.end()) { - symbols.push_back(std::move(ch)); - continue; - } - if (byte_fallback_) { - std::vector bytes; - bool all = true; - for (const unsigned char b : ch) { - char buf[7]; - std::snprintf(buf, sizeof(buf), "<0x%02X>", static_cast(b)); - std::string bt(buf); - if (vocab_.find(bt) == vocab_.end()) { - all = false; - break; + const auto encode_piece = [&](std::string_view piece) { + std::vector symbols; + size_t pos = 0; + while (pos < piece.size()) { + const size_t begin = pos; + (void)DecodeUtf8(piece, pos); + std::string ch(piece.substr(begin, pos - begin)); + if (vocab_.find(ch) != vocab_.end()) { + symbols.push_back(std::move(ch)); + continue; + } + if (byte_fallback_) { + std::vector bytes; + bool all = true; + for (const unsigned char b : ch) { + char buf[7]; + std::snprintf(buf, sizeof(buf), "<0x%02X>", static_cast(b)); + std::string bt(buf); + if (vocab_.find(bt) == vocab_.end()) { + all = false; + break; + } + bytes.push_back(std::move(bt)); + } + if (all) { + for (auto& b : bytes) symbols.push_back(std::move(b)); + continue; } - bytes.push_back(std::move(bt)); } - if (all) { - for (auto& b : bytes) symbols.push_back(std::move(b)); - continue; + if (unk_id_ < 0) { + Fail("SentencePiece: character \"" + std::string(ch) + + "\" has no vocab token, byte-fallback unavailable, and no unk_token"); } + symbols.push_back(kUnk); } - if (unk_id_ < 0) { - Fail("SentencePiece: character \"" + ch + - "\" has no vocab token, byte-fallback unavailable, and no unk_token"); - } - symbols.push_back(kUnk); - } - BpeMerge(symbols, merge_ranks_); + BpeMerge(symbols, merge_ranks_); - // 3) Map merged symbols to ids; fuse consecutive unk ids when fuse_unk. - int32_t prev = -1; - for (const std::string& sym : symbols) { - int32_t id; - if (sym == kUnk) { - id = unk_id_; - } else { - const auto it = vocab_.find(sym); - if (it != vocab_.end()) { - id = it->second; - } else if (unk_id_ >= 0) { + int32_t prev = -1; + for (const std::string& sym : symbols) { + int32_t id; + if (sym == kUnk) { id = unk_id_; } else { - Fail("SentencePiece: merged symbol \"" + sym + "\" not in vocab"); + const auto it = vocab_.find(sym); + if (it != vocab_.end()) { + id = it->second; + } else if (unk_id_ >= 0) { + id = unk_id_; + } else { + Fail("SentencePiece: merged symbol \"" + sym + "\" not in vocab"); + } } + if (fuse_unk_ && id == unk_id_ && prev == unk_id_) continue; + out.push_back(id); + prev = id; } - if (fuse_unk_ && id == unk_id_ && prev == unk_id_) continue; - out.push_back(id); - prev = id; + }; + + if (!metaspace_split_) { + encode_piece(s); + return; + } + // Metaspace `split: true` (HF tokenizers 0.22 pre_tokenizers/metaspace.rs + // `pre_tokenize`, SplitDelimiterBehavior::MergedWithNext): every occurrence + // of the replacement STARTS a new pretoken with the replacement attached to + // what follows, and BPE runs per pretoken, so merges never cross a ▁ + // boundary. Text before the first occurrence (prepend_scheme "never", or a + // non-space-initial segment) is its own leading pretoken. + size_t piece_start = 0; + size_t next = s.find(metaspace_replacement_, + piece_start + (s.compare(0, metaspace_replacement_.size(), + metaspace_replacement_) == 0 + ? metaspace_replacement_.size() + : 0)); + while (next != std::string::npos) { + encode_piece(std::string_view(s).substr(piece_start, next - piece_start)); + piece_start = next; + next = s.find(metaspace_replacement_, next + metaspace_replacement_.size()); } + encode_piece(std::string_view(s).substr(piece_start)); } namespace { @@ -1027,6 +1077,33 @@ std::vector Tokenizer::EncodeWithSpecialTokens( std::string Tokenizer::SpDecodeTokens(const std::vector& tokens, size_t begin, size_t end) const { + if (sp_decoder_metaspace_) { + // The bare `Metaspace` DECODER (HF tokenizers 0.22 pre_tokenizers/ + // metaspace.rs `decode_chain`, reached via decoders/mod.rs): per token, + // each replacement occurrence is DROPPED in the first token (when + // prepend_scheme != "never") and becomes ONE space in every later token; + // all other bytes pass through; tokens concatenate. Deliberately NO + // ByteFallback / Fuse / Strip β€” that is the Sequence chain below, and the + // two differ observably (a first token "▁▁x" decodes to "x" here and to + // " x" there). A window decodes as its own sequence (t == begin is "first"), + // which is exactly how the incremental HF decode of a token slice behaves. + std::string out; + for (size_t t = begin; t < end; ++t) { + const std::string& tk = tokens[t]; + size_t p = 0; + while (p < tk.size()) { + if (tk.compare(p, sp_decoder_replacement_.size(), + sp_decoder_replacement_) == 0) { + if (t != begin || sp_decoder_prepend_never_) out.push_back(' '); + p += sp_decoder_replacement_.size(); + } else { + out.push_back(tk[p]); + ++p; + } + } + } + return out; + } // HF tokenizers Sequence decoder for the SentencePiece family: // Replace(▁->" ") -> ByteFallback -> Fuse -> Strip(1 leading space). // Replace only affects ▁ (never inside a "<0xNN>" byte token), so it is diff --git a/src/vllm/transformers_utils/hf_config.cpp b/src/vllm/transformers_utils/hf_config.cpp index aacf2a464..b5cf702db 100644 --- a/src/vllm/transformers_utils/hf_config.cpp +++ b/src/vllm/transformers_utils/hf_config.cpp @@ -314,6 +314,24 @@ RopeParameters ParseRopeParameters(const nlohmann::json& text, } // namespace +std::vector PeekHfArchitectures(const std::string& path) { + // Non-throwing by contract (see the header): any problem yields {} so the + // caller's ordinary LoadHfConfig path keeps owning every diagnostic. + std::ifstream in(path, std::ios::binary); + if (!in) return {}; + nlohmann::json doc = nlohmann::json::parse(in, /*cb=*/nullptr, + /*allow_exceptions=*/false); + if (doc.is_discarded() || !doc.is_object()) return {}; + const auto it = doc.find("architectures"); + if (it == doc.end() || !it->is_array()) return {}; + std::vector archs; + for (const auto& a : *it) { + if (!a.is_string()) return {}; + archs.push_back(a.get()); + } + return archs; +} + HfConfig LoadHfConfig(const std::string& path) { std::ifstream in(path, std::ios::binary); if (!in) { diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index fa41b7cb9..3bcdb6c38 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -217,6 +217,13 @@ vllm_cpp_add_test(test_qwen27_paged_forward vllm/models/test_qwen27_paged_forwar target_include_directories(test_qwen27_paged_forward PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_unicode_data vllm/test_unicode_data.cpp) vllm_cpp_add_test(test_pretokenizer vllm/test_pretokenizer.cpp) +# Metaspace split=true + Metaspace decoder (Parakeet tokenizer shape) β€” +# ARCH-ONE-SURFACE ROW 1 W1(a); gates against the pre-refactor DecodeIds +# reference on the committed parakeet_e2e fixture tokenizers. +vllm_cpp_add_test(test_tokenizer_metaspace_split + vllm/test_tokenizer_metaspace_split.cpp) +target_compile_definitions(test_tokenizer_metaspace_split PRIVATE + PARAKEET_E2E_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/parakeet_e2e") vllm_cpp_add_test(test_bpe vllm/test_bpe.cpp) vllm_cpp_add_test(test_detokenizer vllm/test_detokenizer.cpp) vllm_cpp_add_test(test_request vllm/test_request.cpp) @@ -365,6 +372,16 @@ vllm_cpp_add_test(test_parakeet_encoder vllm/models/test_parakeet_encoder.cpp) vllm_cpp_add_test(test_parakeet_ctc_engine vllm/models/test_parakeet_ctc_engine.cpp) target_compile_definitions(test_parakeet_ctc_engine PRIVATE PARAKEET_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/parakeet") + +# ARCH-ONE-SURFACE ROW 1 fold gate: the library transcription seam +# (ParakeetTranscriber) byte-identical to the PRE-refactor example pipeline +# AND to the committed goldens captured from the pre-refactor binary; plus the +# registry refuse-by-task contract. Fixture committed +# (scripts/mm/parakeet_e2e_fixture_gen.py). +vllm_cpp_add_test(test_parakeet_transcription_fold + vllm/models/test_parakeet_transcription_fold.cpp) +target_compile_definitions(test_parakeet_transcription_fold PRIVATE + PARAKEET_E2E_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/parakeet_e2e") # P5: the RNN-T / TDT TRANSDUCER gate: the LSTM prediction network, the joint # (with the TDT duration head) and the greedy transducer loop vs a dumped # HuggingFace `ParakeetForRNNT` / `ParakeetForTDT` ORACLE from transformers @@ -616,6 +633,11 @@ target_compile_definitions(test_chat_template PRIVATE # The HTTP api_server smoke test needs the vendored cpp-httplib (gated). if(VLLM_CPP_SERVER) vllm_cpp_add_test(test_openai_api_server vllm/entrypoints/openai/test_api_server.cpp) + # /v1/audio/transcriptions dispatch + socket smoke run against the REAL + # library transcription seam on the committed parakeet_e2e fixture + # (ARCH-ONE-SURFACE ROW 1). + target_compile_definitions(test_openai_api_server PRIVATE + PARAKEET_E2E_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/parakeet_e2e") # M3.6: the OpenAI server CONFORMANCE suite β€” the full API contract exercised # end to end over the REAL cpp-httplib server on an ephemeral port. vllm_cpp_add_test(test_openai_conformance vllm/entrypoints/openai/test_conformance.cpp) @@ -643,6 +665,8 @@ target_include_directories(test_bench PRIVATE ${CMAKE_SOURCE_DIR}/examples/bench # engine via the internal MakeEngineHandle hook (reached under src/). vllm_cpp_add_test(test_capi capi/test_capi.cpp) target_include_directories(test_capi PRIVATE ${CMAKE_SOURCE_DIR}/src) +target_compile_definitions(test_capi PRIVATE + PARAKEET_E2E_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/parakeet_e2e") vllm_cpp_add_test(test_chat_prompt capi/test_chat_prompt.cpp) target_include_directories(test_chat_prompt PRIVATE ${CMAKE_SOURCE_DIR}/src) diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index f1f10e9e6..9f216e999 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -1223,6 +1223,149 @@ TEST_CASE("capi: version and abi-version are exposed") { CHECK(std::string(vllm_version()).size() > 0); CHECK(vllm_abi_version() == VLLM_ABI_VERSION); // The engine-config growth (max_num_batched_tokens / scheduling_policy / - // kv_transfer_config) is ABI v9; the jump-forward toggle is ABI v10. - CHECK(vllm_abi_version() >= 10); + // kv_transfer_config) is ABI v9; the jump-forward toggle is ABI v10; the + // transcription slice (vllm_transcribe) is ABI v11. The >= pin is the one + // check that can catch a WRONG bump: the == VLLM_ABI_VERSION assertions here + // and in test_dlopen compare against the same macro and move with it. + CHECK(vllm_abi_version() >= 11); +} + +// ─── ABI v11: audio transcription (ARCH-ONE-SURFACE ROW 1) ─────────────────── +// The FIRST real-checkpoint load gated through the PUBLIC ABI: vllm_engine_load +// on the committed tiny Parakeet fixtures (tests/vllm/models/fixtures/ +// parakeet_e2e), then vllm_transcribe reproducing the transcript goldens the +// PRE-refactor example binary printed. Everything before this exercised +// vllm_engine_load's bad-path contract only (the severity note in +// .agents/specs/surface-coverage-2026-08-07.md Β§ C-ABI capability coverage). + +namespace { +std::string ParakeetFixture(const char* head) { + return std::string(PARAKEET_E2E_FIXTURE_DIR) + "/" + head; +} +std::string ParakeetWav() { + return std::string(PARAKEET_E2E_FIXTURE_DIR) + "/audio.wav"; +} +} // namespace + +TEST_CASE("capi v11: vllm_transcribe reproduces the pre-refactor goldens") { + struct Golden { + const char* head; + std::vector ids; + const char* text; + }; + const std::vector goldens = { + {"ctc", {3, 4, 3}, "atheat"}, + {"rnnt", + {5, 5, 5, 6, 6, 6, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6}, + "sss on on onssssss on on on on on on on on"}, + }; + for (const Golden& g : goldens) { + CAPTURE(g.head); + vllm_model_params mp = vllm_model_params_default(); + const std::string dir = ParakeetFixture(g.head); + mp.model_path = dir.c_str(); + vllm_engine* eng = nullptr; + REQUIRE(vllm_engine_load(&mp, &eng) == VLLM_OK); + REQUIRE(eng != nullptr); + + vllm_transcription_params tp = vllm_transcription_params_default(); + const std::string wav = ParakeetWav(); + tp.audio_path = wav.c_str(); + vllm_transcription out; + REQUIRE(vllm_transcribe(eng, &tp, &out) == VLLM_OK); + REQUIRE(out.token_ids != nullptr); + const std::vector ids(out.token_ids, + out.token_ids + out.n_token_ids); + CHECK(ids == g.ids); + CHECK(out.has_text == 1); + REQUIRE(out.text != nullptr); + CHECK(std::string(out.text) == g.text); + vllm_transcription_free(&out); + CHECK(out.text == nullptr); // zeroed after free + CHECK(out.token_ids == nullptr); + vllm_transcription_free(&out); // double-free is a safe no-op + vllm_engine_free(eng); + } +} + +TEST_CASE("capi v11: refuse-by-task in both directions") { + // Transcription handle: every text entry point refuses with the actionable + // message instead of crashing on the absent text stack. + vllm_model_params mp = vllm_model_params_default(); + const std::string dir = ParakeetFixture("ctc"); + mp.model_path = dir.c_str(); + vllm_engine* asr = nullptr; + REQUIRE(vllm_engine_load(&mp, &asr) == VLLM_OK); + + vllm_sampling_params sp = vllm_sampling_params_default(); + vllm_completion comp; + CHECK(vllm_complete(asr, "hello", &sp, &comp) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(std::string(vllm_last_error()).find("transcription-only") != + std::string::npos); + char* chat_out = nullptr; + CHECK(vllm_chat(asr, "{\"messages\":[]}", &chat_out) == + VLLM_ERR_INVALID_ARGUMENT); + CHECK(std::string(vllm_last_error()).find("vllm_transcribe") != + std::string::npos); + vllm_engine_free(asr); + + // Text handle: vllm_transcribe refuses symmetrically. + vllm_engine* text = MakeSyntheticEngine(); + REQUIRE(text != nullptr); + vllm_transcription_params tp = vllm_transcription_params_default(); + const std::string wav = ParakeetWav(); + tp.audio_path = wav.c_str(); + vllm_transcription out; + CHECK(vllm_transcribe(text, &tp, &out) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(std::string(vllm_last_error()).find("text-generation engine") != + std::string::npos); + vllm_engine_free(text); +} + +TEST_CASE("capi v11: vllm_transcribe argument contract") { + vllm_model_params mp = vllm_model_params_default(); + const std::string dir = ParakeetFixture("ctc"); + mp.model_path = dir.c_str(); + vllm_engine* eng = nullptr; + REQUIRE(vllm_engine_load(&mp, &eng) == VLLM_OK); + + vllm_transcription out; + // Neither input selected. + vllm_transcription_params none = vllm_transcription_params_default(); + CHECK(vllm_transcribe(eng, &none, &out) == VLLM_ERR_INVALID_ARGUMENT); + // Both inputs selected. + vllm_transcription_params both = vllm_transcription_params_default(); + const std::string wav = ParakeetWav(); + const float pcm[4] = {0.0f, 0.0f, 0.0f, 0.0f}; + both.audio_path = wav.c_str(); + both.pcm = pcm; + both.n_samples = 4; + both.sample_rate = 16000; + CHECK(vllm_transcribe(eng, &both, &out) == VLLM_ERR_INVALID_ARGUMENT); + // pcm without a sample rate. + vllm_transcription_params bad_pcm = vllm_transcription_params_default(); + bad_pcm.pcm = pcm; + bad_pcm.n_samples = 4; + CHECK(vllm_transcribe(eng, &bad_pcm, &out) == VLLM_ERR_INVALID_ARGUMENT); + // Null engine / params / out. + CHECK(vllm_transcribe(nullptr, &none, &out) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_transcribe(eng, nullptr, &out) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_transcribe(eng, &none, nullptr) == VLLM_ERR_INVALID_ARGUMENT); + // Unreadable audio file -> runtime error naming the path problem. + vllm_transcription_params missing = vllm_transcription_params_default(); + missing.audio_path = "/nonexistent/vllm-cpp/audio.wav"; + CHECK(vllm_transcribe(eng, &missing, &out) == VLLM_ERR_RUNTIME); + CHECK(std::string(vllm_last_error()).size() > 0); + // A raw-PCM arm through the ABI marshals and runs end to end. + std::vector silence(4000, 0.0f); + vllm_transcription_params pcm_ok = vllm_transcription_params_default(); + pcm_ok.pcm = silence.data(); + pcm_ok.n_samples = static_cast(silence.size()); + pcm_ok.sample_rate = 16000; + REQUIRE(vllm_transcribe(eng, &pcm_ok, &out) == VLLM_OK); + CHECK(out.has_text == 1); + vllm_transcription_free(&out); + // NULL result free is a no-op. + vllm_transcription_free(nullptr); + vllm_engine_free(eng); } diff --git a/tests/scripts/test_check_surface_coverage.py b/tests/scripts/test_check_surface_coverage.py index 2ae635ee1..a8565a83b 100644 --- a/tests/scripts/test_check_surface_coverage.py +++ b/tests/scripts/test_check_surface_coverage.py @@ -311,9 +311,13 @@ def test_boundary_green(self) -> None: self.assertNotIn("examples/cli", reaching) self.assertNotIn("examples/cli", allow) # The known CLI-only capability drivers ARE internal-reachers. + # (parakeet_transcribe left this list when the ROW 1 fold made it a + # clean vllm.h client β€” asserted below instead.) for unit in ("examples/laguna_gen", "examples/deepseek_v4_gen", - "examples/minimax_h3_gen", "examples/parakeet_transcribe"): + "examples/minimax_h3_gen"): self.assertIn(unit, reaching) + self.assertNotIn("examples/parakeet_transcribe", reaching) + self.assertNotIn("examples/parakeet_transcribe", allow) def test_public_surface_pinned_green(self) -> None: self.assertTrue(mod.public_surface_pinned(mod.read(mod.CMAKELISTS))) @@ -324,10 +328,10 @@ def test_reaching_count_within_ratchet(self) -> None: reaching |= set(mod.internal_include_dir_grant_units(mod.read(mod.EXAMPLES_CMAKE))) self.assertLessEqual(len(reaching), mod.MAX_INTERNAL_REACHING) - def test_ratchet_ceiling_pinned_at_12(self) -> None: - # EQUALITY pin: a ceiling bump (up OR down) must move this line + the "12" claims in + def test_ratchet_ceiling_pinned_at_11(self) -> None: + # EQUALITY pin: a ceiling bump (up OR down) must move this line + the ratchet claims in # the spec/state, so the change is test-visible and reviewed, never silent. - self.assertEqual(mod.MAX_INTERNAL_REACHING, 12) + self.assertEqual(mod.MAX_INTERNAL_REACHING, 11) def test_capability_green(self) -> None: cap_allow, allow_errors = mod.parse_allowlist(mod.read(mod.CAP_ALLOWLIST)) diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index 065bcb354..f0123466d 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -12,6 +12,7 @@ // (tiny hybrid-MoE Qwen3.6 + the BPE fixture, vocab ids 0..21). #include "vllm/entrypoints/openai/api_server.h" #include "vllm/entrypoints/openai/video_api.h" +#include "vllm/multimodal/parakeet_transcription.h" #include @@ -2114,3 +2115,197 @@ TEST_CASE("api_server: the /v1/videos routes do not exist without a runner") { }); } } + +// ─── /v1/audio/transcriptions (ARCH-ONE-SURFACE ROW 1) ─────────────────────── +// Task-conditional like /v1/videos: a TEXT server never registers the route; a +// serving-less (transcription-only) server registers it and NOT the generate +// routes β€” vLLM's supported_tasks-conditional registration +// (api_server.py:255-265) + speech_to_text/transcription semantics. The +// transcriber wraps the REAL library seam (ParakeetTranscriber) on the +// committed parakeet_e2e fixture, so the route is gated against the SAME +// pre-refactor transcript golden as the C ABI and the example. + +namespace { + +std::string ReadFileBytes(const std::string& path) { + std::ifstream f(path, std::ios::binary); + REQUIRE_MESSAGE(f.good(), "cannot open ", path); + return std::string((std::istreambuf_iterator(f)), + std::istreambuf_iterator()); +} + +struct AsrHarness { + vllm::entrypoints::openai::OpenAIServingModels models{"parakeet-fixture"}; + ApiServer server{models, "test-version"}; + std::shared_ptr transcriber; + + AsrHarness() { + transcriber = std::make_shared( + vllm::multimodal::ParakeetTranscriber::FromDir( + std::string(PARAKEET_E2E_FIXTURE_DIR) + "/ctc")); + auto t = transcriber; + server.set_transcriber([t](const uint8_t* wav, size_t n) { + return t->TranscribeWavBytes(wav, n); + }); + } + std::string wav_bytes() const { + return ReadFileBytes(std::string(PARAKEET_E2E_FIXTURE_DIR) + "/audio.wav"); + } +}; + +} // namespace + +TEST_CASE("api_server: transcriptions dispatch reproduces the golden") { + AsrHarness h; + const std::string wav = h.wav_bytes(); + + // Default response_format ("json") -> TranscriptionResponse {"text": ...}. + ApiServer::DispatchResult r = h.server.handle_audio_transcriptions(wav, ""); + CHECK(r.status == 200); + CHECK(r.content_type == "application/json"); + CHECK(json::parse(r.body).at("text") == "atheat"); + + // response_format=text -> the raw transcript as text/plain. + r = h.server.handle_audio_transcriptions(wav, "text"); + CHECK(r.status == 200); + CHECK(r.content_type == "text/plain; charset=utf-8"); + CHECK(r.body == "atheat"); + + // Unsupported formats are named residuals -> 400. + r = h.server.handle_audio_transcriptions(wav, "verbose_json"); + CHECK(r.status == 400); + CHECK(json::parse(r.body).at("error").at("type") == "BadRequestError"); + + // An empty upload -> 400. + r = h.server.handle_audio_transcriptions("", ""); + CHECK(r.status == 400); + + // Undecodable audio -> 400 naming the cause. + r = h.server.handle_audio_transcriptions("not a wav at all", ""); + CHECK(r.status == 400); + + // The serving-less server refuses the generate handlers with the + // NotImplementedError mirror (the socket layer does not even register them). + r = h.server.handle_completions("{}"); + CHECK(r.status == 500); + CHECK(json::parse(r.body).at("error").at("message").get().find( + "does not support Completions") != std::string::npos); + r = h.server.handle_chat_completions("{}"); + CHECK(r.status == 500); +} + +TEST_CASE("api_server: transcriptions without a transcriber is a 500, not a crash") { + vllm::entrypoints::openai::OpenAIServingModels models{"no-asr"}; + ApiServer server{models, "test-version"}; + ApiServer::DispatchResult r = server.handle_audio_transcriptions("bytes", ""); + CHECK(r.status == 500); + CHECK(json::parse(r.body).at("error").at("message").get().find( + "does not support Transcriptions") != std::string::npos); +} + +TEST_CASE("api_server: transcriptions socket smoke (multipart), generate routes 404") { + AsrHarness h; + const int port = h.server.bind_to_any_port("127.0.0.1"); + REQUIRE(port > 0); + std::thread server_thread([&h]() { h.server.serve(); }); + for (int i = 0; i < 500 && !h.server.is_running(); ++i) + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + REQUIRE(h.server.is_running()); + + { + httplib::Client client("127.0.0.1", port); + client.set_connection_timeout(5, 0); + client.set_read_timeout(15, 0); + + // Multipart upload, exactly the OpenAI wire shape. + httplib::UploadFormDataItems items = { + {"file", h.wav_bytes(), "audio.wav", "audio/wav"}, + {"response_format", "json", "", ""}, + }; + auto res = client.Post("/v1/audio/transcriptions", items); + REQUIRE(res); + CHECK(res->status == 200); + CHECK(json::parse(res->body).at("text") == "atheat"); + + // A multipart body without the `file` part -> 400. + httplib::UploadFormDataItems no_file = { + {"response_format", "json", "", ""}, + }; + auto bad = client.Post("/v1/audio/transcriptions", no_file); + REQUIRE(bad); + CHECK(bad->status == 400); + + // The generate routes are NOT registered on a transcription-only server. + auto completions = client.Post("/v1/completions", "{}", "application/json"); + REQUIRE(completions); + CHECK(completions->status == 404); + auto chat = client.Post("/v1/chat/completions", "{}", "application/json"); + REQUIRE(chat); + CHECK(chat->status == 404); + + // Liveness + discovery still serve. + auto health = client.Get("/health"); + REQUIRE(health); + CHECK(health->status == 200); + auto models_res = client.Get("/v1/models"); + REQUIRE(models_res); + CHECK(models_res->status == 200); + CHECK(json::parse(models_res->body).at("data").at(0).at("id") == + "parakeet-fixture"); + } + + h.server.stop(); + server_thread.join(); +} + +TEST_CASE("api_server: the audio routes do not exist on a TEXT server") { + // The reverse of the ASR socket smoke above, and the exact twin of "the + // /v1/videos routes do not exist without a runner": task-conditional + // registration means a TEXT-engine server (no transcriber attached) must + // answer 404 from the ROUTE TABLE for /v1/audio/*. This pins the + // `if (transcriber_)` registration gate itself β€” the direct-dispatch 500 + // test above cannot see route registration, so `if (true)` there would + // register the route on every text server and only THIS test reds (the + // mutated server answers 400/500 from the handler instead of 404). + const HfConfig c = MakeConfig(); + const Qwen3_5MoeWeights w = MakeWeights(c); + ServerHarness h(c, w, Fixture()); + + const int port = h.server.bind_to_any_port("127.0.0.1"); + REQUIRE(port > 0); + std::thread server_thread([&h]() { h.server.serve(); }); + for (int i = 0; i < 500 && !h.server.is_running(); ++i) + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + REQUIRE(h.server.is_running()); + + { + httplib::Client client("127.0.0.1", port); + client.set_connection_timeout(5, 0); + client.set_read_timeout(15, 0); + + // A well-formed multipart upload β€” exactly what the route would accept if + // it existed β€” must fall through to httplib's 404, proving the route was + // never registered (not merely that the handler rejected the payload). + httplib::UploadFormDataItems items = { + {"file", "RIFF fake", "audio.wav", "audio/wav"}, + }; + auto res = client.Post("/v1/audio/transcriptions", items); + REQUIRE(res); + CHECK(res->status == 404); + + // /v1/audio/translations is NOT routed anywhere yet (a named residual of + // the ROW 1 fold): 404 on the text server documents that absence too. + auto translations = client.Post("/v1/audio/translations", items); + REQUIRE(translations); + CHECK(translations->status == 404); + + // The text server still serves its own task, so the 404s above are about + // the audio routes, not a dead server. + auto health = client.Get("/health"); + REQUIRE(health); + CHECK(health->status == 200); + } + + h.server.stop(); + server_thread.join(); +} diff --git a/tests/vllm/models/fixtures/parakeet_e2e/audio.wav b/tests/vllm/models/fixtures/parakeet_e2e/audio.wav new file mode 100644 index 000000000..15188ee4b Binary files /dev/null and b/tests/vllm/models/fixtures/parakeet_e2e/audio.wav differ diff --git a/tests/vllm/models/fixtures/parakeet_e2e/ctc/config.json b/tests/vllm/models/fixtures/parakeet_e2e/ctc/config.json new file mode 100644 index 000000000..6c289b10c --- /dev/null +++ b/tests/vllm/models/fixtures/parakeet_e2e/ctc/config.json @@ -0,0 +1,24 @@ +{ + "architectures": [ + "ParakeetForCTC" + ], + "model_type": "parakeet_ctc", + "vocab_size": 8, + "pad_token_id": 7, + "encoder_config": { + "hidden_size": 16, + "num_hidden_layers": 1, + "num_attention_heads": 2, + "intermediate_size": 32, + "num_mel_bins": 8, + "subsampling_conv_channels": 4, + "subsampling_factor": 4, + "subsampling_conv_kernel_size": 3, + "subsampling_conv_stride": 2, + "conv_kernel_size": 3, + "attention_bias": true, + "convolution_bias": true, + "scale_input": true, + "hidden_act": "silu" + } +} diff --git a/tests/vllm/models/fixtures/parakeet_e2e/ctc/model.safetensors b/tests/vllm/models/fixtures/parakeet_e2e/ctc/model.safetensors new file mode 100644 index 000000000..91d58a243 Binary files /dev/null and b/tests/vllm/models/fixtures/parakeet_e2e/ctc/model.safetensors differ diff --git a/tests/vllm/models/fixtures/parakeet_e2e/ctc/tokenizer.json b/tests/vllm/models/fixtures/parakeet_e2e/ctc/tokenizer.json new file mode 100644 index 000000000..cda6c242e --- /dev/null +++ b/tests/vllm/models/fixtures/parakeet_e2e/ctc/tokenizer.json @@ -0,0 +1,36 @@ +{ + "version": "1.0", + "pre_tokenizer": { + "type": "Metaspace", + "replacement": "▁", + "prepend_scheme": "always", + "split": true + }, + "decoder": { + "type": "Metaspace", + "replacement": "▁", + "prepend_scheme": "always", + "split": true + }, + "model": { + "type": "BPE", + "unk_token": null, + "vocab": { + "▁the": 0, + "▁cat": 1, + "▁sat": 2, + "at": 3, + "he": 4, + "s": 5, + "▁on": 6 + }, + "merges": [] + }, + "added_tokens": [ + { + "id": 7, + "content": "", + "special": true + } + ] +} diff --git a/tests/vllm/models/fixtures/parakeet_e2e/golden_ctc.txt b/tests/vllm/models/fixtures/parakeet_e2e/golden_ctc.txt new file mode 100644 index 000000000..6b18781b8 --- /dev/null +++ b/tests/vllm/models/fixtures/parakeet_e2e/golden_ctc.txt @@ -0,0 +1,2 @@ +3 4 3 +atheat diff --git a/tests/vllm/models/fixtures/parakeet_e2e/golden_rnnt.txt b/tests/vllm/models/fixtures/parakeet_e2e/golden_rnnt.txt new file mode 100644 index 000000000..03a5ce2d8 --- /dev/null +++ b/tests/vllm/models/fixtures/parakeet_e2e/golden_rnnt.txt @@ -0,0 +1,2 @@ +5 5 5 6 6 6 5 5 5 5 5 5 6 6 6 6 6 6 6 6 +sss on on onssssss on on on on on on on on diff --git a/tests/vllm/models/fixtures/parakeet_e2e/rnnt/config.json b/tests/vllm/models/fixtures/parakeet_e2e/rnnt/config.json new file mode 100644 index 000000000..e86c9e4f9 --- /dev/null +++ b/tests/vllm/models/fixtures/parakeet_e2e/rnnt/config.json @@ -0,0 +1,29 @@ +{ + "architectures": [ + "ParakeetForRNNT" + ], + "model_type": "parakeet_rnnt", + "vocab_size": 8, + "blank_token_id": 7, + "pad_token_id": 2, + "decoder_hidden_size": 8, + "num_decoder_layers": 1, + "max_symbols_per_step": 3, + "hidden_act": "relu", + "encoder_config": { + "hidden_size": 16, + "num_hidden_layers": 1, + "num_attention_heads": 2, + "intermediate_size": 32, + "num_mel_bins": 8, + "subsampling_conv_channels": 4, + "subsampling_factor": 4, + "subsampling_conv_kernel_size": 3, + "subsampling_conv_stride": 2, + "conv_kernel_size": 3, + "attention_bias": true, + "convolution_bias": true, + "scale_input": true, + "hidden_act": "silu" + } +} diff --git a/tests/vllm/models/fixtures/parakeet_e2e/rnnt/generation_config.json b/tests/vllm/models/fixtures/parakeet_e2e/rnnt/generation_config.json new file mode 100644 index 000000000..4a900f4e2 --- /dev/null +++ b/tests/vllm/models/fixtures/parakeet_e2e/rnnt/generation_config.json @@ -0,0 +1,3 @@ +{ + "decoder_start_token_id": 7 +} diff --git a/tests/vllm/models/fixtures/parakeet_e2e/rnnt/model.safetensors b/tests/vllm/models/fixtures/parakeet_e2e/rnnt/model.safetensors new file mode 100644 index 000000000..c74f95267 Binary files /dev/null and b/tests/vllm/models/fixtures/parakeet_e2e/rnnt/model.safetensors differ diff --git a/tests/vllm/models/fixtures/parakeet_e2e/rnnt/tokenizer.json b/tests/vllm/models/fixtures/parakeet_e2e/rnnt/tokenizer.json new file mode 100644 index 000000000..cda6c242e --- /dev/null +++ b/tests/vllm/models/fixtures/parakeet_e2e/rnnt/tokenizer.json @@ -0,0 +1,36 @@ +{ + "version": "1.0", + "pre_tokenizer": { + "type": "Metaspace", + "replacement": "▁", + "prepend_scheme": "always", + "split": true + }, + "decoder": { + "type": "Metaspace", + "replacement": "▁", + "prepend_scheme": "always", + "split": true + }, + "model": { + "type": "BPE", + "unk_token": null, + "vocab": { + "▁the": 0, + "▁cat": 1, + "▁sat": 2, + "at": 3, + "he": 4, + "s": 5, + "▁on": 6 + }, + "merges": [] + }, + "added_tokens": [ + { + "id": 7, + "content": "", + "special": true + } + ] +} diff --git a/tests/vllm/models/test_model_registry.cpp b/tests/vllm/models/test_model_registry.cpp index 266a8c73f..e21d7fef7 100644 --- a/tests/vllm/models/test_model_registry.cpp +++ b/tests/vllm/models/test_model_registry.cpp @@ -44,7 +44,9 @@ HfConfig Config(std::vector architectures) { TEST_CASE("registry_imports: every registered architecture has a complete factory") { const auto registrations = ModelRegistry::Registrations(); - REQUIRE(registrations.size() == 30); + // 30 text archs + the 3 Parakeet transcription-only archs (ARCH-ONE-SURFACE + // ROW 1: ParakeetForCTC/ForRNNT/ForTDT, SupportsTranscription mirror). + REQUIRE(registrations.size() == 33); for (const ModelRegistration& registration : registrations) { CAPTURE(registration.architecture); @@ -136,7 +138,7 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { // with the kExampleConfigArchitectures ledger; adding a model appends its two // entries here. const std::vector supported = ModelRegistry::SupportedArchs(); - REQUIRE(supported.size() == 30); + REQUIRE(supported.size() == 33); CHECK(std::is_sorted(supported.begin(), supported.end())); // The full byte-order sequence. Note "MiniCPM3" < "MiniCPMF" and "Phi3" < // "PhiF" ('3' 0x33 < 'F' 0x46); "OPT" < "Olmo" ('P' 0x50 < 'l' 0x6C); and among @@ -165,6 +167,9 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { "OPTForCausalLM", "Olmo2ForCausalLM", "Olmo3ForCausalLM", + "ParakeetForCTC", + "ParakeetForRNNT", + "ParakeetForTDT", "Phi3ForCausalLM", "PhiForCausalLM", "Qwen3ForCausalLM", @@ -189,7 +194,22 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { TEST_CASE("registry_model_property: Qwen registrations match pinned _ModelInfo") { for (const ModelRegistration& registration : ModelRegistry::Registrations()) { CAPTURE(registration.architecture); + if (registration.architecture.substr(0, 8) == "Parakeet") { + // ARCH-ONE-SURFACE ROW 1: the ASR family mirrors SupportsTranscription + // (interfaces.py:1110-1118) β€” transcription-capable, transcription-ONLY, + // NOT a text-generation model; the text entrypoints refuse it by task. + CHECK_FALSE(registration.info.is_text_generation_model); + CHECK(registration.info.supports_transcription); + CHECK(registration.info.supports_transcription_only); + CHECK_FALSE(registration.info.is_pooling_model); + CHECK_FALSE(registration.info.is_hybrid); + CHECK_FALSE(registration.info.supports_multimodal); + continue; + } CHECK(registration.info.is_text_generation_model); + // No text arch is transcription-capable (SupportsTranscription default). + CHECK_FALSE(registration.info.supports_transcription); + CHECK_FALSE(registration.info.supports_transcription_only); CHECK_FALSE(registration.info.is_pooling_model); CHECK_FALSE(registration.info.has_inner_state); CHECK(registration.info.score_type == "bi-encoder"); @@ -532,7 +552,7 @@ TEST_CASE("Qwen3.5 SSM cache dtype accepts upstream torch aliases exactly") { TEST_CASE("hf_registry_coverage: every registration has an example config fixture") { // C++ fixture registry for the currently implemented subset. Keep this list // alias-for-alias with the central ordered table, mirroring HF_EXAMPLE_MODELS. - constexpr std::array kExampleConfigArchitectures{ + constexpr std::array kExampleConfigArchitectures{ "CohereForCausalLM", "DeepseekV2ForCausalLM", "DeepseekV4ForCausalLM", @@ -555,6 +575,9 @@ TEST_CASE("hf_registry_coverage: every registration has an example config fixtur "OPTForCausalLM", "Olmo2ForCausalLM", "Olmo3ForCausalLM", + "ParakeetForCTC", + "ParakeetForRNNT", + "ParakeetForTDT", "Phi3ForCausalLM", "PhiForCausalLM", "Qwen3ForCausalLM", @@ -643,6 +666,7 @@ TEST_CASE("raise_for_unsupported: subset default message and order match oracle" "'LlamaForCausalLM', " "'MiniCPM3ForCausalLM', 'MiniCPMForCausalLM', 'MistralForCausalLM', " "'OPTForCausalLM', 'Olmo2ForCausalLM', 'Olmo3ForCausalLM', " + "'ParakeetForCTC', 'ParakeetForRNNT', 'ParakeetForTDT', " "'Phi3ForCausalLM', 'PhiForCausalLM', 'Qwen3ForCausalLM', " "'Qwen3MoeForCausalLM', 'Qwen3VLForConditionalGeneration', " "'Qwen3_5ForConditionalGeneration', " @@ -664,6 +688,7 @@ TEST_CASE("raise_for_unsupported: subset default message and order match oracle" "'LlamaForCausalLM', " "'MiniCPM3ForCausalLM', 'MiniCPMForCausalLM', 'MistralForCausalLM', " "'OPTForCausalLM', 'Olmo2ForCausalLM', 'Olmo3ForCausalLM', " + "'ParakeetForCTC', 'ParakeetForRNNT', 'ParakeetForTDT', " "'Phi3ForCausalLM', 'PhiForCausalLM', 'Qwen3ForCausalLM', " "'Qwen3MoeForCausalLM', 'Qwen3VLForConditionalGeneration', " "'Qwen3_5ForConditionalGeneration', " diff --git a/tests/vllm/models/test_parakeet_transcription_fold.cpp b/tests/vllm/models/test_parakeet_transcription_fold.cpp new file mode 100644 index 000000000..f44ee115f --- /dev/null +++ b/tests/vllm/models/test_parakeet_transcription_fold.cpp @@ -0,0 +1,268 @@ +// ARCH-ONE-SURFACE ROW 1 fold gate: the library transcription seam +// (vllm::multimodal::ParakeetTranscriber) must reproduce the EXACT ids and +// transcript the PRE-refactor examples/parakeet_transcribe binary produced. +// +// Three arms, all on the committed deterministic fixtures +// (tests/vllm/models/fixtures/parakeet_e2e, scripts/mm/ +// parakeet_e2e_fixture_gen.py): +// 1. OLD-vs-NEW byte identity: the pre-refactor pipeline is replicated here +// verbatim (main.cpp @ f98e1e48: ReadWav16BitMono :49-99, head dispatch +// :210-238, LoadVocab :104-126 + DecodeIds :140-159) and must agree with +// the seam id-for-id and byte-for-byte on text. +// 2. GOLDEN identity: both must equal the COMMITTED golden files captured by +// running the actual pre-refactor binary (golden_ctc.txt / +// golden_rnnt.txt) β€” so the in-test replica cannot drift into agreeing +// with a subtly changed library. +// 3. Refuse-by-task: the registered Parakeet archs resolve from config.json +// with the SupportsTranscription-only ModelInfo, and the text-generation +// entry (LoadedEngine::FromModelDir) refuses them with the actionable +// message instead of crashing (interfaces.py:1110-1118 mirror). +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "doctest/doctest.h" +#include "vllm/entrypoints/model_loader.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/model_executor/models/parakeet_encoder.h" +#include "vllm/model_executor/models/parakeet_transducer.h" +#include "vllm/multimodal/parakeet_audio_processor.h" +#include "vllm/multimodal/parakeet_transcription.h" +#include "vt/backend.h" + +namespace { + +std::string Fix() { return std::string(PARAKEET_E2E_FIXTURE_DIR); } + +// ── the PRE-refactor example pipeline, replicated verbatim ────────────────── +// examples/parakeet_transcribe/main.cpp:49-99 @ f98e1e48. +bool RefReadWav16BitMono(const std::string& path, std::vector* out, + int* sample_rate) { + std::ifstream f(path, std::ios::binary); + if (!f) return false; + char riff[12]; + f.read(riff, 12); + if (std::memcmp(riff, "RIFF", 4) != 0 || + std::memcmp(riff + 8, "WAVE", 4) != 0) { + return false; + } + int channels = 0; + int bits = 0; + while (f) { + char id[4]; + uint32_t sz = 0; + f.read(id, 4); + f.read(reinterpret_cast(&sz), 4); + if (!f) break; + if (std::memcmp(id, "fmt ", 4) == 0) { + std::vector fmt(sz); + f.read(fmt.data(), static_cast(sz)); + uint16_t ch = 0, bps = 0; + uint32_t sr = 0; + std::memcpy(&ch, fmt.data() + 2, 2); + std::memcpy(&sr, fmt.data() + 4, 4); + std::memcpy(&bps, fmt.data() + 14, 2); + channels = ch; + bits = bps; + *sample_rate = static_cast(sr); + } else if (std::memcmp(id, "data", 4) == 0) { + if (channels != 1 || bits != 16) return false; + const size_t n = sz / 2; + std::vector pcm(n); + f.read(reinterpret_cast(pcm.data()), static_cast(sz)); + out->resize(n); + for (size_t i = 0; i < n; ++i) { + (*out)[i] = static_cast(pcm[i]) / 32768.0F; + } + return true; + } else { + f.seekg(sz, std::ios::cur); + } + } + return false; +} + +// main.cpp:104-126 @ f98e1e48. +std::map RefLoadVocab(const std::string& dir) { + std::map vocab; + std::ifstream f(dir + "/tokenizer.json", std::ios::binary); + if (!f.good()) return vocab; + nlohmann::json doc; + f >> doc; + const auto model = doc.find("model"); + if (model != doc.end()) { + const auto v = model->find("vocab"); + if (v != model->end() && v->is_object()) { + for (auto it = v->begin(); it != v->end(); ++it) { + vocab[it.value().get()] = it.key(); + } + } + } + const auto added = doc.find("added_tokens"); + if (added != doc.end() && added->is_array()) { + for (const auto& t : *added) { + vocab[t.at("id").get()] = t.at("content").get(); + } + } + return vocab; +} + +// main.cpp:140-159 @ f98e1e48. +std::string RefDecodeIds(const std::vector& ids, + const std::map& vocab) { + static const std::string kReplacement = "\xe2\x96\x81"; + std::string text; + for (size_t i = 0; i < ids.size(); ++i) { + const auto it = vocab.find(ids[i]); + if (it == vocab.end()) continue; + const std::string& piece = it->second; + for (size_t p = 0; p < piece.size();) { + if (piece.compare(p, kReplacement.size(), kReplacement) == 0) { + if (i != 0) text.push_back(' '); + p += kReplacement.size(); + } else { + text.push_back(piece[p]); + ++p; + } + } + } + return text; +} + +// The old pipeline end to end (main.cpp:176-264): WAV -> features -> the head +// config.json names -> ids -> text. +struct OldResult { + std::vector ids; + std::string text; +}; + +OldResult RunOldPipeline(const std::string& ckpt, const std::string& wav) { + namespace mm = vllm::multimodal; + OldResult r; + std::vector samples; + int sample_rate = 0; + REQUIRE(RefReadWav16BitMono(wav, &samples, &sample_rate)); + + const std::string model_type = mm::LoadParakeetModelType(ckpt); + const mm::ParakeetEncoderConfig probe = mm::LoadParakeetConfig(ckpt); + mm::ParakeetExtractorConfig ecfg; + ecfg.feature_size = static_cast(probe.num_mel_bins); + const mm::ParakeetAudioProcessor proc(ecfg); + const mm::ParakeetAudioFeatures feats = proc.ProcessWaveform( + samples.data(), static_cast(samples.size()), sample_rate); + + vt::Backend& cpu = vt::GetBackend(vt::DeviceType::kCPU); + if (model_type == "parakeet_rnnt" || model_type == "parakeet_tdt") { + mm::ParakeetEncoderConfig enc_cfg; + mm::ParakeetTransducerConfig cfg; + const mm::ParakeetForTransducerWeights w = + mm::LoadParakeetTransducer(ckpt, &enc_cfg, &cfg); + r.ids = mm::ParakeetForTransducerForward(feats.input_features, + feats.num_frames, + feats.valid_frames, w, enc_cfg, + cfg, cpu) + .token_ids; + } else { + mm::ParakeetEncoderConfig cfg; + const mm::ParakeetForCTCWeights w = mm::LoadParakeetForCTC(ckpt, &cfg); + r.ids = mm::ParakeetForCTCForward(feats.input_features, feats.num_frames, + feats.valid_frames, w, cfg, cpu) + .token_ids; + } + r.text = RefDecodeIds(r.ids, RefLoadVocab(ckpt)); + return r; +} + +// Parse a committed golden file: line 1 = space-joined ids, line 2 = text. +OldResult ReadGolden(const std::string& path) { + std::ifstream f(path, std::ios::binary); + REQUIRE_MESSAGE(f.good(), "cannot open golden: ", path); + OldResult g; + std::string ids_line; + REQUIRE(static_cast(std::getline(f, ids_line))); + std::istringstream is(ids_line); + int32_t id = 0; + while (is >> id) g.ids.push_back(id); + REQUIRE(static_cast(std::getline(f, g.text))); + return g; +} + +} // namespace + +TEST_CASE("fold gate: seam == pre-refactor pipeline == committed golden") { + const std::string wav = Fix() + "/audio.wav"; + for (const char* head : {"ctc", "rnnt"}) { + CAPTURE(head); + const std::string ckpt = Fix() + "/" + head; + + const OldResult old_r = RunOldPipeline(ckpt, wav); + const OldResult golden = ReadGolden(Fix() + "/golden_" + head + ".txt"); + + const auto t = vllm::multimodal::ParakeetTranscriber::FromDir(ckpt); + CHECK(t.has_tokenizer()); + const vllm::multimodal::ParakeetTranscription got = + t.TranscribeWavFile(wav); + + // Old vs new, byte-identical. + CHECK(got.token_ids == old_r.ids); + CHECK(got.has_text); + CHECK(got.text == old_r.text); + // Both vs the transcript the pre-refactor BINARY printed. + CHECK(old_r.ids == golden.ids); + CHECK(old_r.text == golden.text); + CHECK(got.token_ids == golden.ids); + CHECK(got.text == golden.text); + } +} + +TEST_CASE("fold gate: PCM entry equals WAV entry") { + // The seam's raw-PCM entry (the C-ABI `pcm` arm) must agree with the WAV + // path on the same samples. + const std::string wav = Fix() + "/audio.wav"; + std::vector samples; + int rate = 0; + REQUIRE(RefReadWav16BitMono(wav, &samples, &rate)); + const auto t = + vllm::multimodal::ParakeetTranscriber::FromDir(Fix() + "/ctc"); + const auto via_wav = t.TranscribeWavFile(wav); + const auto via_pcm = + t.Transcribe(samples.data(), static_cast(samples.size()), rate); + CHECK(via_pcm.token_ids == via_wav.token_ids); + CHECK(via_pcm.text == via_wav.text); +} + +TEST_CASE("registry: Parakeet archs resolve as transcription-only") { + for (const char* arch : + {"ParakeetForCTC", "ParakeetForRNNT", "ParakeetForTDT"}) { + CAPTURE(arch); + const std::vector archs = {arch}; + const vllm::ModelRegistration& reg = vllm::ModelRegistry::Resolve( + std::span(archs)); + CHECK(reg.info.supports_transcription); + CHECK(reg.info.supports_transcription_only); + CHECK_FALSE(reg.info.is_text_generation_model); + } +} + +TEST_CASE("refuse-by-task: the text engine refuses a Parakeet dir cleanly") { + // LoadedEngine::FromModelDir must throw the actionable transcription-only + // message (never a crash, never a deep loader error about missing text + // tensors). + const std::string ckpt = Fix() + "/ctc"; + try { + vllm::entrypoints::EngineParams params; + (void)vllm::entrypoints::LoadedEngine::FromModelDir(ckpt, params); + FAIL("FromModelDir was expected to refuse a transcription-only arch"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + MESSAGE("refusal message: ", msg); + CHECK(msg.find("supports transcription only") != std::string::npos); + CHECK(msg.find("/v1/audio/transcriptions") != std::string::npos); + } +} diff --git a/tests/vllm/test_model_loader_gguf.cpp b/tests/vllm/test_model_loader_gguf.cpp index 24978efc1..fe6cf0e97 100644 --- a/tests/vllm/test_model_loader_gguf.cpp +++ b/tests/vllm/test_model_loader_gguf.cpp @@ -85,6 +85,7 @@ TEST_CASE("FromModelDir rejects an unknown dense architecture before loading") { "'LlamaForCausalLM', " "'MiniCPM3ForCausalLM', 'MiniCPMForCausalLM', 'MistralForCausalLM', " "'OPTForCausalLM', 'Olmo2ForCausalLM', 'Olmo3ForCausalLM', " + "'ParakeetForCTC', 'ParakeetForRNNT', 'ParakeetForTDT', " "'Phi3ForCausalLM', 'PhiForCausalLM', 'Qwen3ForCausalLM', " "'Qwen3MoeForCausalLM', 'Qwen3VLForConditionalGeneration', " "'Qwen3_5ForConditionalGeneration', " diff --git a/tests/vllm/test_tokenizer_metaspace_split.cpp b/tests/vllm/test_tokenizer_metaspace_split.cpp new file mode 100644 index 000000000..24748b4d4 --- /dev/null +++ b/tests/vllm/test_tokenizer_metaspace_split.cpp @@ -0,0 +1,239 @@ +// Metaspace `split: true` + the Metaspace DECODER in vllm::Tokenizer β€” +// ARCH-ONE-SURFACE ROW 1 (Parakeet ASR fold), W1(a). +// +// Ported from HF tokenizers 0.22: +// - pre_tokenizers/metaspace.rs `Metaspace::pre_tokenize` + its test module +// (`basic`, `multiple_spaces`): with `split: true` the normalized string is +// split at every replacement occurrence with SplitDelimiterBehavior:: +// MergedWithNext (the ▁ starts a NEW pretoken, attached to what follows), +// so BPE merges can never cross a ▁ boundary. +// - decoders (metaspace.rs `decode_chain` + the decoder test): inside the +// FIRST token every replacement char is DROPPED (when prepend_scheme != +// "never"), inside every later token it becomes ONE space; tokens are +// concatenated. No ByteFallback / Fuse / Strip β€” the bare Metaspace decoder +// is NOT the Mistral/Gemma Sequence chain. +// +// Why now: every published Parakeet checkpoint ships `{"type": "Metaspace", +// "replacement": "▁", "prepend_scheme": "always", "split": true}` as BOTH +// pre_tokenizer and decoder. `Tokenizer::FromHfJson` used to refuse split=true +// ("no golden in scope"), which is exactly why examples/parakeet_transcribe +// grew a private LoadVocab + DecodeIds pair. This suite is the golden that +// guard asked for; the reference arm below replicates the pre-refactor +// DecodeIds byte for byte and pins the library equal to it on the committed +// pre-refactor transcript goldens. +#include +#include +#include +#include +#include +#include +#include + +#include "doctest/doctest.h" +#include "vllm/tokenizer/tokenizer.h" + +namespace { + +using vllm::tok::Tokenizer; + +// Write `body` to a fresh temp tokenizer.json and load it. +Tokenizer FromJson(const std::string& body) { + static int counter = 0; + const std::string path = + (std::filesystem::temp_directory_path() / + ("metaspace_split_tok_" + std::to_string(counter++) + ".json")) + .string(); + { + std::ofstream out(path, std::ios::binary); + out << body; + } + Tokenizer tok = Tokenizer::FromHfJson(path); + std::remove(path.c_str()); + return tok; +} + +// A minimal SentencePiece-family tokenizer.json. `split_flag` goes into the +// Metaspace pre_tokenizer; `decoder` (when non-empty) is spliced verbatim. +std::string MakeJson(const std::string& vocab, const std::string& merges, + bool split_flag, const std::string& decoder, + const std::string& prepend = "always") { + std::string s = "{"; + s += "\"pre_tokenizer\":{\"type\":\"Metaspace\",\"replacement\":\"▁\"," + "\"prepend_scheme\":\"" + prepend + "\",\"split\":"; + s += split_flag ? "true" : "false"; + s += "},"; + if (!decoder.empty()) s += "\"decoder\":" + decoder + ","; + s += "\"model\":{\"type\":\"BPE\",\"vocab\":" + vocab + + ",\"merges\":" + merges + "}}"; + return s; +} + +const std::string kMetaspaceDecoder = + "{\"type\":\"Metaspace\",\"replacement\":\"▁\"," + "\"prepend_scheme\":\"always\",\"split\":true}"; + +} // namespace + +TEST_CASE("metaspace split=true is accepted (the old guard is gone)") { + // Pre-change this threw `Metaspace split=true unsupported (no golden in + // scope)`; the Parakeet fold implements the rule instead of refusing. + const Tokenizer tok = FromJson(MakeJson( + R"({"▁":0,"a":1,"▁a":2})", "[]", /*split=*/true, "")); + CHECK(tok.IsSentencePiece()); +} + +TEST_CASE("split=true pre-splits at the replacement: merges cannot cross") { + // tokenizers 0.22 pre_tokenizers/metaspace.rs `basic`, expressed at the + // encode observable: "a a" normalizes to ▁a▁a; MergedWithNext yields the + // pretokens [▁a][▁a], so the ▁a+▁a merge can NEVER apply under split=true + // and MUST apply under split=false. Same file otherwise. + const std::string vocab = + R"({"▁":0,"a":1,"▁a":2,"▁a▁a":3})"; + const std::string merges = + R"([["▁","a"],["▁a","▁a"]])"; + const Tokenizer split_true = + FromJson(MakeJson(vocab, merges, /*split=*/true, "")); + CHECK(split_true.Encode("a a") == std::vector{2, 2}); + const Tokenizer split_false = + FromJson(MakeJson(vocab, merges, /*split=*/false, "")); + CHECK(split_false.Encode("a a") == std::vector{3}); +} + +TEST_CASE("split=true keeps consecutive spaces as separate ▁ pretokens") { + // tokenizers 0.22 metaspace.rs `multiple_spaces`: "a a" -> ▁a ▁ ▁a. + const std::string vocab = + R"({"▁":0,"a":1,"▁a":2,"▁▁":3})"; + const std::string merges = + R"([["▁","a"],["▁","▁"]])"; + const Tokenizer tok = FromJson(MakeJson(vocab, merges, /*split=*/true, "")); + // Under split=true the ▁+▁ merge cannot apply either (each ▁ starts its own + // pretoken), so the middle space stays a lone ▁ piece. + CHECK(tok.Encode("a a") == std::vector{2, 0, 2}); +} + +TEST_CASE("Metaspace decoder: first piece drops ▁, later pieces map ▁ to space") { + // tokenizers 0.22 decoders test: decode_chain(["▁Hey", "▁friend!"]) == + // ["Hey", " friend!"], i.e. "Hey friend!" concatenated. + const std::string vocab = + R"({"▁Hey":0,"▁friend!":1,"▁▁x":2})"; + const Tokenizer tok = + FromJson(MakeJson(vocab, "[]", /*split=*/true, kMetaspaceDecoder)); + CHECK(tok.Decode({0, 1}) == "Hey friend!"); + CHECK(tok.Decode({1}) == "friend!"); + // EVERY replacement inside the first piece is dropped, not just a leading + // one β€” that is the flat_map rule, and it is what distinguishes the + // Metaspace decoder from the Sequence chain's Strip(1 leading space). + CHECK(tok.Decode({2}) == "x"); + CHECK(tok.Decode({0, 2}) == "Hey x"); + CHECK(tok.Decode({}).empty()); +} + +TEST_CASE("Metaspace decoder with prepend_scheme=never keeps the space") { + const std::string never_decoder = + "{\"type\":\"Metaspace\",\"replacement\":\"▁\"," + "\"prepend_scheme\":\"never\",\"split\":true}"; + const std::string vocab = R"({"▁Hey":0,"▁friend!":1})"; + const Tokenizer tok = FromJson( + MakeJson(vocab, "[]", /*split=*/true, never_decoder, "never")); + CHECK(tok.Decode({0, 1}) == " Hey friend!"); +} + +TEST_CASE("no Metaspace decoder node: the Sequence chain is unchanged") { + // Regression pin for Mistral/Gemma: a file WITHOUT a Metaspace decoder keeps + // the Sequence decoder (Replace -> ByteFallback -> Fuse -> Strip ONE leading + // space). "▁▁x" is the distinguishing probe: Sequence gives " x" (two spaces, + // one stripped), the Metaspace decoder gives "x" (both dropped in piece 0). + const std::string vocab = R"({"▁Hey":0,"▁▁x":1})"; + const Tokenizer tok = FromJson(MakeJson(vocab, "[]", /*split=*/false, "")); + CHECK(tok.Decode({1}) == " x"); + CHECK(tok.Decode({0}) == "Hey"); +} + +// ── the Parakeet fixture: the library now decodes what the example decoded ── + +#include + +namespace { + +// The PRE-refactor examples/parakeet_transcribe reference, replicated byte for +// byte (main.cpp:104-159 @ f98e1e48, LoadVocab + DecodeIds): id -> piece from +// model.vocab + added_tokens, then the Metaspace decode_chain rule applied by +// hand β€” exactly the code the split=true guard forced the example to carry. +std::map RefLoadVocabJson(const std::string& dir) { + std::map vocab; + std::ifstream f(dir + "/tokenizer.json", std::ios::binary); + REQUIRE(f.good()); + nlohmann::json doc; + f >> doc; + const auto model = doc.find("model"); + if (model != doc.end()) { + const auto v = model->find("vocab"); + if (v != model->end() && v->is_object()) { + for (auto it = v->begin(); it != v->end(); ++it) { + vocab[it.value().get()] = it.key(); + } + } + } + const auto added = doc.find("added_tokens"); + if (added != doc.end() && added->is_array()) { + for (const auto& t : *added) { + vocab[t.at("id").get()] = t.at("content").get(); + } + } + return vocab; +} + +std::string RefDecodeIds(const std::vector& ids, + const std::map& vocab) { + static const std::string kReplacement = "\xe2\x96\x81"; // U+2581 + std::string text; + for (size_t i = 0; i < ids.size(); ++i) { + const auto it = vocab.find(ids[i]); + if (it == vocab.end()) continue; + const std::string& piece = it->second; + for (size_t p = 0; p < piece.size();) { + if (piece.compare(p, kReplacement.size(), kReplacement) == 0) { + if (i != 0) text.push_back(' '); + p += kReplacement.size(); + } else { + text.push_back(piece[p]); + ++p; + } + } + } + return text; +} + +std::string FixtureDir() { return std::string(PARAKEET_E2E_FIXTURE_DIR); } + +} // namespace + +TEST_CASE("parakeet fixture tokenizer: Tokenizer::Decode == pre-refactor DecodeIds") { + for (const char* head : {"ctc", "rnnt"}) { + const std::string dir = FixtureDir() + "/" + head; + const Tokenizer tok = Tokenizer::FromHfJson(dir + "/tokenizer.json"); + const std::map ref_vocab = RefLoadVocabJson(dir); + // The committed pre-refactor golden id sequences, plus probes that hit the + // first-piece rule and the added token. + const std::vector> cases = { + {3, 4, 3}, // golden_ctc ids + {5, 5, 5, 6, 6, 6, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6}, // rnnt + {0, 3, 6}, // ▁the at ▁on β€” first-piece drop + later-piece space + {6}, // lone ▁on as the first piece + {7}, // the added decodes literally + {}, + }; + for (const auto& ids : cases) { + CHECK_MESSAGE(tok.Decode(ids) == RefDecodeIds(ids, ref_vocab), + "head=", head, " n_ids=", ids.size()); + } + // The two committed transcript goldens, verbatim. + if (std::string(head) == "ctc") { + CHECK(tok.Decode({3, 4, 3}) == "atheat"); + } else { + CHECK(tok.Decode({5, 5, 5, 6, 6, 6, 5, 5, 5, 5, 5, 5, + 6, 6, 6, 6, 6, 6, 6, 6}) == + "sss on on onssssss on on on on on on on on"); + } + } +}