Skip to content

dots3-note W7c-1: accept a multi-channel WAV at 16 kHz by mean-reducing it, and stop naming librosa as the oracle #2813

Description

@localai-org-maint-bot

Row: MODEL-MM-dots3-note-dots3-note-for-causal-lm

W7c-1 is the CHANNELS half of W7c, split out because it is an exact mirror with no oracle
risk, while the sample-rate half is an unavoidable divergence. Keeping them apart stops one
claim borrowing the other's credibility.

What is refused today, and why it is nearly free to lift

The audio path accepts PCM16 mono WAV at 16 kHz. A stereo or multi-channel PCM16 WAV
already at 16 kHz is refused only for want of a mean over channels.

Upstream reduces to mono in two places, both plain means:

  • decode side, vllm/multimodal/media/audio.py:207-208 @ 9035151d6np.mean over channels
    when mono=True, which is load_audio's default (:220);
  • parser side, vllm/multimodal/audio.py:46-56 and :91-161 @ 9035151d6
    ChannelReduction.MEAN is the default and AudioSpec.target_channels is 1, which
    common/processor.py:523-525 selects for dots3.

So this is a mirror, roughly ten lines, and it stops refusing a whole class of real files.

Scope

Widen DecodeWavPcm16Mono (src/vllm/multimodal/audio_processor.cpp:35-79), or add a
channel-aware sibling beside it, to accept multi-channel PCM16 at the target rate and reduce
by mean. Nothing else changes.

Reachability

Production entry point ApiServer::handle_chat_completionsInstallMultiModalChatSeam
(server_main.cpp:1565) → MakeDots3NoteChatSeamRouteDots3NoteAudioWav
(mm_chat_dots3note.cpp:210).

The smallest failing test already exists and must invert. The SUBCASE
"a STEREO WAV names the container refusal and W7c" at
tests/vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp:1379 asserts the refusal
today. It must go true-before / false-after, which is the ownership test AGENTS.md asks for.
Replace it with a case asserting the stereo request SERVES and that its logprobs equal those
of an independently computed per-sample mean.

In flow, in the same change: three statements about the oracle are false

librosa occurs exactly ONCE at 9035151d6, in a comment at vllm/multimodal/audio.py:31
about get_duration. It is not on the decode or resample path at all. The real chain is
soundfile/libsndfile first, PyAV/ffmpeg as fallback, and libswresample for the resample.

Three places in this tree say otherwise, and two of them are production error strings a
user reads
:

  • src/vllm/entrypoints/openai/mm_chat_dots3note.cpp:233-234 — "Upstream decodes with
    librosa through its data parser"
  • src/vllm/multimodal/dots3_note_processor.cpp:526-528 — "upstream resamples in its data
    parser ... with librosa"
  • .agents/specs/dots3-note.md ## Owed — "accepts whatever librosa can open"

Correct all three. Also stale in that same ## Owed entry: "a windowed-sinc resampler is a
numerically delicate port of its own" — true as difficulty, but it is no longer a blocker,
since #2583 landed Ltx2ResampleWaveform.

Not in scope

The sample-rate arm (W7c-2, a recorded divergence). The container arm, which is leaving this
row entirely — it needs a vendored demuxer, libsndfile alone reports 26 formats, and four
other places already want the same capability.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions