Skip to content

[Feature] Add ARK-ASR-3B support as an ASR model #1142

Description

@Peterren

Checklist

  • This is a feature proposal, not a usage question.
  • This issue is written in English.

Motivation

ARK-ASR-3B (AutoArk-AI, Apache-2.0) is a multilingual open ASR model that maps cleanly onto SGLang-Omni's existing single-stage batched ASR pipeline, reusing the native Qwen2 decoder. Its model card reports state-of-the-art results on the Hugging Face Open ASR Leaderboard English short-form benchmark (avg WER 5.04%, RTFx 490.98 across AMI, Earnings22, GigaSpeech, LibriSpeech, SPGISpeech, VoxPopuli), and it supports ~19 languages.

Its architecture is structurally the same as the ASR models already served here (Qwen3-ASR, MOSS-Transcribe-Diarize): a Whisper-style audio tower -> MLP frame-merge adapter -> dense LLM, with audio embeddings scattered into placeholder positions.

Audio (16kHz) -> WhisperFeatureExtractor -> mel [B, 128, T]
  -> WhisperEncoder (32L, d=1280, RoPE self-attention)
  -> MLP adapter (Linear->GELU->Linear + LayerNorm; 4x frame merge)
  -> scatter into <|audio|> (id 151663) positions
  -> Qwen2 LLM (36L, hidden=2048, GQA 16/2 heads, tied embeddings) -> transcript
Component Details
Encoder WhisperSpecialEncoder -- 32L, d_model=1280, 20 heads, RoPE self-attention (not absolute positions)
Adapter AudioMLPAdapter -- 2-layer MLP + LayerNorm, merge_factor=4
LLM dense Qwen2ForCausalLM -- 36L, hidden=2048, GQA (16 q / 2 kv), tied embeddings
Injection embedding scatter at `<
Config architectures: ["ArkasrForConditionalGeneration"], model_type: arkasr

This complements the existing Qwen3-ASR / Fun-ASR / MOSS-TD ASR paths (roadmap #924) with another open ASR family, exercising the same pipeline concepts.

One detail worth flagging: the checkpoint's shipped remote code targets the transformers-4 WhisperEncoderLayer API (encoder_layer(...)[0]), which is incompatible with the transformers-5 layer contract this repo uses -- so a native reimplementation of the audio tower is required rather than running the remote code, the same approach the higgs_audio_asr adapter took.

Scope

Conservative, model-local, following the qwen3_asr + higgs_audio_asr convention:

  • sglang_omni/models/arkasr/ scaffold: native config/processor, natively reimplemented RoPE-Whisper audio tower + MLP adapter, ArkasrForConditionalGeneration reusing SGLang's native Qwen2ForCausalLM, request builder (mel + prompt template + audio-token scatter), and a single-stage PipelineConfig.
  • Register ArkasrForConditionalGeneration in the model runner + arch->config map; register the native config with AutoConfig so trust_remote_code is not needed at serve time.
  • CPU-light unit tests for config registration, stage defaults, audio-token-count formula, and tower forward shape.

Status

A working implementation is ready; PR opened alongside this issue. Validation against the official transformers checkpoint on SeedTTS EN:

  • The audio encoder reimplementation is numerically equivalent to the reference (max abs diff 0.0 on identical mel inputs across multiple lengths).
  • All checkpoint weights load with 0 missing / 0 unexpected tensors.
  • End-to-end output matches the official model on clean-EN clips: 37/37 clips byte-identical; corpus WER 0.0000 (5-clip alignment set), 0.0101 (200 clips), 0.0113 (full 1088-clip SeedTTS EN).
  • Throughput on a single A100 worker: ~6.4 / 16 / 31 / 37 / 39 samples/s at concurrency 1 / 4 / 16 / 32 / 64.

Happy to adjust the integration shape (prompt template surface, task-instruction handling, DP/MPS wiring) to match maintainer preference.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions