Skip to content

Add support for adapter switching and configurable encoders - #106

Merged
aviv1ron1 merged 4 commits into
asrfrom
asr-switch-v2
Jul 21, 2026
Merged

Add support for adapter switching and configurable encoders#106
aviv1ron1 merged 4 commits into
asrfrom
asr-switch-v2

Conversation

@aviv1ron1

Copy link
Copy Markdown
Collaborator

What this adds

Speech input via a cascade, running as a single vLLM model — no separate speech server, no change to how the model is deployed or called. Audio is transcribed to text by an ASR encoder and the transcript tokens are spliced into the prompt before the decoder runs; the LLM only ever sees text.

The ASR encoder is distil-whisper/distil-small.en by default, but any HuggingFace automatic-speech-recognition pipeline model works via config alone (e.g. openai/whisper-small for multilingual) — no code change to swap it.

Capabilities

  • Single-model cascade — audio in through the standard OpenAI/chat API (input_audio content parts); the chat template emits the <|audio|> marker and the processor replaces it with the real transcript tokens.
  • Audio + adapters — audio requests route through the switch exactly like text, so an audio request can activate a LoRA adapter identically to its text equivalent.
  • Config-carried ASR tuning — asr_pipeline_kwargs (pipeline construction, e.g. chunk_length_s) and asr_generate_kwargs (decode defaults, e.g. language/task) baked into config.json so the checkpoint is self-describing.
  • Per-request language — a request may override decode kwargs via mm_processor_kwargs (allowlisted to language/task).
  • Multiple audio clips per request — configurable ceiling (asr_max_audio_clips, default 32), each spliced at its own marker.
  • Context-derived transcript budget — replaces the old fixed cap; per-clip budget = (max_model_len − generation_reserve − prompt) ÷ clips.
  • Encoder-agnostic long-audio chunker — asr_self_chunks=false routes long audio through split → per-window transcribe → overlap-dedup merge, so a fixed-window backend inherits long-audio support (Whisper keeps its own internal chunking).
  • Gated per checkpoint by asr_enabled — a non-audio checkpoint reports no audio modality and never loads ASR.

All of the above are settable at compose time (matching --asr-* flags) and editable directly in config.json.

Set requires_raw_input_tokens=True so vLLM passes raw input_ids to forward on the
multimodal path, letting the switch detect adapter control tokens for audio
requests. embed_input_ids now applies the switch's token-exchange rewrite
(control -> substitute id) via a new SingleSwitch.apply_token_exchange helper, so
control tokens get in-distribution embeddings exactly as on the text path.
Remove the two hard caps in the audio cascade and add an encoder-agnostic
long-audio chunker:

1. Multiple clips per request. get_supported_mm_limits is now a configurable
   ceiling (asr_max_audio_clips, default 32) instead of a hardcoded 1; each
   clip's transcript is spliced at its own <|audio|> marker (the splice loop
   already supported N). For the cascade the clips cost no extra KV (transcripts
   are ordinary text tokens bounded by the context); the ceiling exists to bound
   the per-request synchronous-ASR work and the startup profiling pass.

2. Context-derived transcript budget. The fixed 2048-token cap is replaced by
   audio_token_budget() = (context - generation_reserve - prompt) / clips,
   floored at 1. get_mm_max_tokens_per_item derives from the seq_len vLLM passes;
   _call_hf_processor reads the served max_model_len and subtracts the actual
   prompt. New asr_generation_reserve_tokens (default 8192).

3. Encoder-agnostic chunker (chunking.py): split_waveform + merge_transcripts
   (word-level overlap de-dup). ASRTranscriber.transcribe gains a self_chunks
   flag: Whisper (self_chunks=True) keeps its internal timestamp stitching;
   backends with a fixed input window route through our split/transcribe/merge.
   New config: asr_self_chunks, asr_chunk_length_s, asr_chunk_overlap_s.

Compose CLI flags and docs/AUDIO.md updated. Tests: new test_chunking.py; budget
and chunked-path tests in test_asr.py; config round-trip/validation in
test_config.py; multi-clip, budget, and flag-forwarding in test_audio_processor.py.

Already-composed checkpoints inherit the new behavior via getattr defaults; no
re-compose required (only self_chunks=False needs a config change).
@aviv1ron1
aviv1ron1 marked this pull request as ready for review July 21, 2026 14:54
@aviv1ron1
aviv1ron1 merged commit 7796f58 into asr Jul 21, 2026
@aviv1ron1
aviv1ron1 deleted the asr-switch-v2 branch July 21, 2026 14:55
@aviv1ron1
aviv1ron1 restored the asr-switch-v2 branch July 22, 2026 23:01
@aviv1ron1
aviv1ron1 deleted the asr-switch-v2 branch July 27, 2026 08:39
aviv1ron1 added a commit that referenced this pull request Jul 27, 2026
Add support for adapter switching and configurable encoders

Signed-off-by: Aviv Ron <rona@il.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant