Checklist
Motivation
ARK-ASR-3B (AutoArk-AI, Apache-2.0) is a multilingual open ASR model that maps cleanly onto SGLang-Omni's existing single-stage batched ASR pipeline, reusing the native Qwen2 decoder. Its model card reports state-of-the-art results on the Hugging Face Open ASR Leaderboard English short-form benchmark (avg WER 5.04%, RTFx 490.98 across AMI, Earnings22, GigaSpeech, LibriSpeech, SPGISpeech, VoxPopuli), and it supports ~19 languages.
Its architecture is structurally the same as the ASR models already served here (Qwen3-ASR, MOSS-Transcribe-Diarize): a Whisper-style audio tower -> MLP frame-merge adapter -> dense LLM, with audio embeddings scattered into placeholder positions.
Audio (16kHz) -> WhisperFeatureExtractor -> mel [B, 128, T]
-> WhisperEncoder (32L, d=1280, RoPE self-attention)
-> MLP adapter (Linear->GELU->Linear + LayerNorm; 4x frame merge)
-> scatter into <|audio|> (id 151663) positions
-> Qwen2 LLM (36L, hidden=2048, GQA 16/2 heads, tied embeddings) -> transcript
| Component |
Details |
| Encoder |
WhisperSpecialEncoder -- 32L, d_model=1280, 20 heads, RoPE self-attention (not absolute positions) |
| Adapter |
AudioMLPAdapter -- 2-layer MLP + LayerNorm, merge_factor=4 |
| LLM |
dense Qwen2ForCausalLM -- 36L, hidden=2048, GQA (16 q / 2 kv), tied embeddings |
| Injection |
embedding scatter at `< |
| Config |
architectures: ["ArkasrForConditionalGeneration"], model_type: arkasr |
This complements the existing Qwen3-ASR / Fun-ASR / MOSS-TD ASR paths (roadmap #924) with another open ASR family, exercising the same pipeline concepts.
One detail worth flagging: the checkpoint's shipped remote code targets the transformers-4 WhisperEncoderLayer API (encoder_layer(...)[0]), which is incompatible with the transformers-5 layer contract this repo uses -- so a native reimplementation of the audio tower is required rather than running the remote code, the same approach the higgs_audio_asr adapter took.
Scope
Conservative, model-local, following the qwen3_asr + higgs_audio_asr convention:
sglang_omni/models/arkasr/ scaffold: native config/processor, natively reimplemented RoPE-Whisper audio tower + MLP adapter, ArkasrForConditionalGeneration reusing SGLang's native Qwen2ForCausalLM, request builder (mel + prompt template + audio-token scatter), and a single-stage PipelineConfig.
- Register
ArkasrForConditionalGeneration in the model runner + arch->config map; register the native config with AutoConfig so trust_remote_code is not needed at serve time.
- CPU-light unit tests for config registration, stage defaults, audio-token-count formula, and tower forward shape.
Status
A working implementation is ready; PR opened alongside this issue. Validation against the official transformers checkpoint on SeedTTS EN:
- The audio encoder reimplementation is numerically equivalent to the reference (max abs diff 0.0 on identical mel inputs across multiple lengths).
- All checkpoint weights load with 0 missing / 0 unexpected tensors.
- End-to-end output matches the official model on clean-EN clips: 37/37 clips byte-identical; corpus WER 0.0000 (5-clip alignment set), 0.0101 (200 clips), 0.0113 (full 1088-clip SeedTTS EN).
- Throughput on a single A100 worker: ~6.4 / 16 / 31 / 37 / 39 samples/s at concurrency 1 / 4 / 16 / 32 / 64.
Happy to adjust the integration shape (prompt template surface, task-instruction handling, DP/MPS wiring) to match maintainer preference.
Checklist
Motivation
ARK-ASR-3B (AutoArk-AI, Apache-2.0) is a multilingual open ASR model that maps cleanly onto SGLang-Omni's existing single-stage batched ASR pipeline, reusing the native Qwen2 decoder. Its model card reports state-of-the-art results on the Hugging Face Open ASR Leaderboard English short-form benchmark (avg WER 5.04%, RTFx 490.98 across AMI, Earnings22, GigaSpeech, LibriSpeech, SPGISpeech, VoxPopuli), and it supports ~19 languages.
Its architecture is structurally the same as the ASR models already served here (Qwen3-ASR, MOSS-Transcribe-Diarize): a Whisper-style audio tower -> MLP frame-merge adapter -> dense LLM, with audio embeddings scattered into placeholder positions.
WhisperSpecialEncoder-- 32L, d_model=1280, 20 heads, RoPE self-attention (not absolute positions)AudioMLPAdapter-- 2-layer MLP + LayerNorm,merge_factor=4Qwen2ForCausalLM-- 36L, hidden=2048, GQA (16 q / 2 kv), tied embeddingsarchitectures: ["ArkasrForConditionalGeneration"],model_type: arkasrThis complements the existing Qwen3-ASR / Fun-ASR / MOSS-TD ASR paths (roadmap #924) with another open ASR family, exercising the same pipeline concepts.
One detail worth flagging: the checkpoint's shipped remote code targets the transformers-4
WhisperEncoderLayerAPI (encoder_layer(...)[0]), which is incompatible with the transformers-5 layer contract this repo uses -- so a native reimplementation of the audio tower is required rather than running the remote code, the same approach the higgs_audio_asr adapter took.Scope
Conservative, model-local, following the
qwen3_asr+higgs_audio_asrconvention:sglang_omni/models/arkasr/scaffold: native config/processor, natively reimplemented RoPE-Whisper audio tower + MLP adapter,ArkasrForConditionalGenerationreusing SGLang's nativeQwen2ForCausalLM, request builder (mel + prompt template + audio-token scatter), and a single-stagePipelineConfig.ArkasrForConditionalGenerationin the model runner + arch->config map; register the native config withAutoConfigsotrust_remote_codeis not needed at serve time.Status
A working implementation is ready; PR opened alongside this issue. Validation against the official
transformerscheckpoint on SeedTTS EN:Happy to adjust the integration shape (prompt template surface, task-instruction handling, DP/MPS wiring) to match maintainer preference.