Fix two audio prompt-placeholder invariant violations (multi-modal #51, #52)

[aviv1ron1]

Fixes three defects in the ASR audio cascade, plus the packaging gap that blocked testing them.

1. _hf_processor_applies_updates contract violation (#51)

Left at its base value, the hook returns True for raw items — telling vLLM the HF processor already expanded <|audio|>. vLLM then skips our PromptReplacement and searches for transcript ids that aren't there, since _call_hf_processor deliberately leaves the marker in place and returns the transcript out of band. Result: RuntimeError: Expected there to be 1 audio prompt placeholders ... but instead found 0.

Fix: override to return False — we never expand the marker ourselves.

Only the uncached path consults the hook (the cached path hardcodes False), so with the default cache on the bug is invisible. It needs cache-off and a string prompt, which startup profiling passes — so the sharpest symptom is the engine failing to boot at --mm-processor-cache-gb 0. The default path is byte-identical after this change.

2. Empty/silent transcript breaks the placeholder invariant (#52)

Silence, music, or a 200 ms clip transcribes to "" → a zero-length replacement, which vLLM discards before reporting the item missing, so a well-formed request became an opaque 500. Fires on both paths at any cache setting. PlaceholderRange(offset, length) cannot express length=0, and the item cannot be dropped either — its count comes from the parsed request.

Fix: _EMPTY_TRANSCRIPT_TEXT = " ", applied in _transcribe rather than replacement() so it also reaches audio_num_tokens; raises clearly if even the fallback tokenizes to nothing. Only a cascade hits this — a real encoder's placeholder length tracks duration and is never zero.

3. control_to_substitute_lut length drift

Found triaging a CUDA indexSelectSmallIndex: srcIndex < srcSelectDimSize assert. SingleSwitch sizes the table from config.vocab_size, but compose builds the model from the base config's value and grows the vocabulary only afterwards — so --enable-audio shipped a checkpoint whose table was one row short of its own config.json. Unrecoverable at load: from_pretrained discards the mismatched tensor and leaves uninitialised memory, so every id reads as a control id and the rewrite gathers out of bounds. Latent before this branch, where compose ends with vocab_size == max_ctrl_id + 1 and the two computations agree by coincidence.

Fix:

• Rebuild after the resize; sizing extracted to build_control_to_substitute_lut() so __init__ and the rebuild cannot diverge.
• validator.validate_control_lut() asserts len(lut) == config.vocab_size at compose time.
• add_audio_token no longer evicts the control tokens: add_special_tokens replaces additional_special_tokens instead of extending it, so enabling audio silently dropped them from the saved tokenizer_config.json. Token ids unchanged.

4. Missing audio extra

audio (soundfile, librosa) was declared and referenced by nothing — not dev, dev-vllm20, test, or either workflow; integration tests died on ModuleNotFoundError: No module named 'soundfile'. Now requested by dev / dev-vllm20 (test inherits) and stated explicitly in gpu-tests.yaml; docs/AUDIO.md gains an Installing section. ci.yaml untouched — the CPU tier installs no vLLM.

Tests

• tests/vllm/test_audio_processor.py — 38 (was 33), driving vLLM's real _apply_hf_processor_text_mm / _maybe_apply_prompt_updates; CPU-only.
• test_token_exchange.py, test_validator.py, test_tokenizer_setup.py — LUT sizing, rebuild-after-resize, strict save/load round trip, compose-time assertion, special-token preservation.
• tests/integration/test_audio_uncached_processor.py (new) — the only test exercising the uncached path §1 fixes. Boots with the cache disabled (version-detecting mm_processor_cache_gb vs the older disable_mm_preprocessor_cache) and asserts the marker is gone from prompt_token_ids. The cache check gates the fixture, so the module skips rather than silently passing on the cached path.
• audio marker added: pytest -m audio selects 132 tests. Removed test_answerability_over_audio.py — it asserted a non-deterministic model verdict.

§1, §2 and the §3 rebuild were each confirmed as genuine guards by reverting them in turn.

Verification

CPU: 158 unit, 565 hf, 154 composer, all pre-commit hooks, DCO.

GPU, pytest -m audio on both vLLM 0.19 and 0.20 — 132 passed. This includes the new uncached-path module, so §1 is verified end-to-end on both versions: the engine boots with the processor cache disabled and the marker is replaced.

GPU, full suite on 1362bac: 1490 passed, 1 failed, 0 errors — was 1440 / 1 / 31. Zero MISMATCH or device-side asserts, and the CUDA-assert test now passes. The one failure was the answerability test since removed.

Not verified: audio under non-eager (every audio test sets enforce_eager=True) and under TP>1.