Add support for adapter switching and configurable encoders - #106
Merged
Conversation
Set requires_raw_input_tokens=True so vLLM passes raw input_ids to forward on the multimodal path, letting the switch detect adapter control tokens for audio requests. embed_input_ids now applies the switch's token-exchange rewrite (control -> substitute id) via a new SingleSwitch.apply_token_exchange helper, so control tokens get in-distribution embeddings exactly as on the text path.
…errides in config
Remove the two hard caps in the audio cascade and add an encoder-agnostic long-audio chunker: 1. Multiple clips per request. get_supported_mm_limits is now a configurable ceiling (asr_max_audio_clips, default 32) instead of a hardcoded 1; each clip's transcript is spliced at its own <|audio|> marker (the splice loop already supported N). For the cascade the clips cost no extra KV (transcripts are ordinary text tokens bounded by the context); the ceiling exists to bound the per-request synchronous-ASR work and the startup profiling pass. 2. Context-derived transcript budget. The fixed 2048-token cap is replaced by audio_token_budget() = (context - generation_reserve - prompt) / clips, floored at 1. get_mm_max_tokens_per_item derives from the seq_len vLLM passes; _call_hf_processor reads the served max_model_len and subtracts the actual prompt. New asr_generation_reserve_tokens (default 8192). 3. Encoder-agnostic chunker (chunking.py): split_waveform + merge_transcripts (word-level overlap de-dup). ASRTranscriber.transcribe gains a self_chunks flag: Whisper (self_chunks=True) keeps its internal timestamp stitching; backends with a fixed input window route through our split/transcribe/merge. New config: asr_self_chunks, asr_chunk_length_s, asr_chunk_overlap_s. Compose CLI flags and docs/AUDIO.md updated. Tests: new test_chunking.py; budget and chunked-path tests in test_asr.py; config round-trip/validation in test_config.py; multi-clip, budget, and flag-forwarding in test_audio_processor.py. Already-composed checkpoints inherit the new behavior via getattr defaults; no re-compose required (only self_chunks=False needs a config change).
aviv1ron1
marked this pull request as ready for review
July 21, 2026 14:54
aviv1ron1
requested review from
antonpibm,
freunda and
yairallouche
as code owners
July 21, 2026 14:54
aviv1ron1
added a commit
that referenced
this pull request
Jul 27, 2026
Add support for adapter switching and configurable encoders Signed-off-by: Aviv Ron <rona@il.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Speech input via a cascade, running as a single vLLM model — no separate speech server, no change to how the model is deployed or called. Audio is transcribed to text by an ASR encoder and the transcript tokens are spliced into the prompt before the decoder runs; the LLM only ever sees text.
The ASR encoder is distil-whisper/distil-small.en by default, but any HuggingFace automatic-speech-recognition pipeline model works via config alone (e.g. openai/whisper-small for multilingual) — no code change to swap it.
Capabilities
All of the above are settable at compose time (matching --asr-* flags) and editable directly in config.json.