Skip to content

audio: splice full transcript; reject oversized requests via context - #109

Merged
barvhaim merged 1 commit into
asrfrom
bugfix/45-audio-reject-oversized
Jul 27, 2026
Merged

audio: splice full transcript; reject oversized requests via context#109
barvhaim merged 1 commit into
asrfrom
bugfix/45-audio-reject-oversized

Conversation

@barvhaim

Copy link
Copy Markdown
Collaborator

…check (#45)

The audio cascade truncated the transcript to a context-derived per-clip budget (asr_generation_reserve_tokens held back for the answer). When that reserve met or exceeded the served max_model_len, the budget floored to 1 token: the model saw no question and refused, with no error (#45).

Rather than patch the truncation math, remove it. The transcript is now spliced into the prompt in full, as ordinary text tokens. A request whose prompt + transcript(s) can't leave room for the answer within max_model_len is rejected by vLLM's standard prompt-length check (HTTP 400) — the same loud failure text-only requests already get, instead of a silent 1-token transcript. This also makes audio behave identically to long text.

Details:

  • Remove asr_generation_reserve_tokens entirely (config field + validation, compose CLI flag, processor accessor). It only existed to size the old truncation budget.
  • Drop the budget param / ids[:budget] truncation in the processor; splice the full transcript.
  • get_mm_max_tokens_per_item now reports max(1, seq_len // count) — the honest worst-case transcript positions one clip can occupy. This still sizes vLLM's encoder cache correctly (embed_multimodal emits one row per transcript token); it is a profiling hint, not a request bound.
  • Remove the now-dead audio_token_budget helper and its tests.

Tests updated; docs/AUDIO.md rewritten to describe reject-not-truncate.

…check (#45)

The audio cascade truncated the transcript to a context-derived per-clip
budget (asr_generation_reserve_tokens held back for the answer). When that
reserve met or exceeded the served max_model_len, the budget floored to 1
token: the model saw no question and refused, with no error (#45).

Rather than patch the truncation math, remove it. The transcript is now
spliced into the prompt in full, as ordinary text tokens. A request whose
prompt + transcript(s) can't leave room for the answer within max_model_len
is rejected by vLLM's standard prompt-length check (HTTP 400) — the same
loud failure text-only requests already get, instead of a silent 1-token
transcript. This also makes audio behave identically to long text.

Details:
- Remove asr_generation_reserve_tokens entirely (config field + validation,
  compose CLI flag, processor accessor). It only existed to size the old
  truncation budget.
- Drop the budget param / ids[:budget] truncation in the processor; splice
  the full transcript.
- get_mm_max_tokens_per_item now reports max(1, seq_len // count) — the
  honest worst-case transcript positions one clip can occupy. This still
  sizes vLLM's encoder cache correctly (embed_multimodal emits one row per
  transcript token); it is a profiling hint, not a request bound.
- Remove the now-dead audio_token_budget helper and its tests.

Tests updated; docs/AUDIO.md rewritten to describe reject-not-truncate.
@barvhaim
barvhaim merged commit b128676 into asr Jul 27, 2026
@barvhaim
barvhaim deleted the bugfix/45-audio-reject-oversized branch July 27, 2026 11:06
aviv1ron1 pushed a commit that referenced this pull request Jul 27, 2026
…check (#45) (#109)

The audio cascade truncated the transcript to a context-derived per-clip
budget (asr_generation_reserve_tokens held back for the answer). When that
reserve met or exceeded the served max_model_len, the budget floored to 1
token: the model saw no question and refused, with no error (#45).

Rather than patch the truncation math, remove it. The transcript is now
spliced into the prompt in full, as ordinary text tokens. A request whose
prompt + transcript(s) can't leave room for the answer within max_model_len
is rejected by vLLM's standard prompt-length check (HTTP 400) — the same
loud failure text-only requests already get, instead of a silent 1-token
transcript. This also makes audio behave identically to long text.

Details:
- Remove asr_generation_reserve_tokens entirely (config field + validation,
  compose CLI flag, processor accessor). It only existed to size the old
  truncation budget.
- Drop the budget param / ids[:budget] truncation in the processor; splice
  the full transcript.
- get_mm_max_tokens_per_item now reports max(1, seq_len // count) — the
  honest worst-case transcript positions one clip can occupy. This still
  sizes vLLM's encoder cache correctly (embed_multimodal emits one row per
  transcript token); it is a profiling hint, not a request bound.
- Remove the now-dead audio_token_budget helper and its tests.

Tests updated; docs/AUDIO.md rewritten to describe reject-not-truncate.

Co-authored-by: aviv ron <rona@il.ibm.com>

Signed-off-by: Bar Haim <barha@il.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants