qwen3-asr-server

OpenAI-compatible HTTP server for Qwen3-ASR-1.7B (Whisper-style speech-to-text) with vLLM acceleration, fp8 KV cache, context-primed transcription, and clean text output (language prefix auto-stripped).

Built for low-latency conversational voice agents — runs comfortably alongside a TTS model on a single 16 GB GPU and supports 52 languages.

Companion project: qwen3-tts-server — the matching text-to-speech server. Together they form a complete voice loop.

What you get

🎙️ /v1/audio/transcriptions — Whisper-compatible transcription (clean output, no language X\n prefix)
🔄 /v1/audio/translations — API-compatible shim (transcribes in detected language; see note below)
🧠 /v1/chat/completions — context-primed transcription via input_audio parts (vocabulary biasing)
🌍 52 languages — English, French, Chinese, Japanese, Korean, Spanish, German, Italian, Arabic, …
⚡ vLLM backend with fp8 KV cache + prefix caching → ~250 ms for 5 s clip
🧰 Drop-in Whisper replacement — same API, just change the base URL
🐳 Single-container deploy, Ubuntu 24.04, CUDA 12.8 (driver ≥ 520)
🐌 CPU fallback (--cpu) for smoke tests

Architecture: proxy, not execv

Previous versions used os.execv to hand off to qwen-asr-serve, which prevented any response post-processing. The server now starts vLLM as a subprocess on an internal port and proxies all requests through — enabling:

Language prefix stripping: Qwen3-ASR internally prepends language English\n (or language French\n, …) to every response. The server strips this before returning.
Proper /health endpoint that reflects actual model readiness.
Clean /v1/audio/translations shim for Whisper API compatibility.
Extra CLI flags forwarded to vLLM (--gpu-memory-utilization, --kv-cache-dtype, etc.)

Host prerequisites (GPU)

Before running the container, install the NVIDIA driver and container toolkit on the host.

# --- NVIDIA driver (Ubuntu 24.04, from the official CUDA repo) ---
# Open kernel module — recommended for Turing, Ampere, Ada Lovelace, Blackwell
curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub \
  | sudo gpg --dearmor -o /etc/apt/keyrings/nvidia-cuda.gpg
printf 'Types: deb\nURIs: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/\nSuites: /\nSigned-By: /etc/apt/keyrings/nvidia-cuda.gpg\n' \
  | sudo tee /etc/apt/sources.list.d/nvidia-cuda.sources
sudo apt-get update
sudo apt-get install -y nvidia-driver-open nvidia-container-toolkit

# For Docker — configure runtime and restart
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# For Podman — generate CDI specs (enables --device nvidia.com/gpu=all)
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

# Verify
nvidia-smi

Driver note: nvidia-driver-open uses the open kernel module and works on Turing+. For Pascal and older GPUs use the proprietary variant (nvidia-driver-XXX). vGPU / cloud instances: the hypervisor locks the supported guest driver version — use whatever version the cloud provider pre-installs (e.g. 550 on Vultr GRID A100D).

Quickstart (Docker / Podman)

# 1. Run the container
docker run -d --name qwen3-asr \
  --gpus all \
  -p 8002:8000 \
  -v qwen3-hf-cache:/root/.cache/huggingface \
  ghcr.io/malaiwah/qwen3-asr-server:latest \
  --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization 0.65 \
  --max-model-len 4096 \
  --max-num-seqs 4 \
  --kv-cache-dtype fp8

# Podman equivalent (requires CDI — see Host prerequisites above):
# podman run -d --name qwen3-asr \
#   --device nvidia.com/gpu=all \
#   ...

# 2. Watch startup
docker logs -f qwen3-asr   # look for "✓ vLLM backend ready"

# 3. Transcribe
curl -X POST http://localhost:8002/v1/audio/transcriptions \
  -F model=Qwen/Qwen3-ASR-1.7B \
  -F file=@my-recording.wav
# → {"text": "Hello from Qwen3."}

Sharing a GPU with TTS?
Start TTS first (fixed ~4.4 GB footprint), then ASR with --gpu-memory-utilization 0.55.
vLLM auto-sizes its KV cache to whatever VRAM is left.
See docker-compose.yml for the full orchestrated setup.

Quickstart (uv, no container)

git clone https://github.com/malaiwah/qwen3-asr-server.git
cd qwen3-asr-server
uv venv && source .venv/bin/activate

# GPU (vLLM-backed, production):
uv pip install -e ".[gpu]"
python server.py --host 0.0.0.0 --port 8002 \
  --gpu-memory-utilization 0.65 \
  --max-model-len 4096 \
  --kv-cache-dtype fp8

# CPU fallback (transformers, slow):
uv pip install -e ".[cpu]"
python server.py --cpu --port 8002

# In another shell:
./test-asr.py my-recording.wav
./test-asr.py my-recording.wav --language English
./test-asr.py my-recording.wav --context "Hermes Agent, Honcho memory, oikos"

Note on GPU install: vLLM may require nvcc (CUDA toolkit) to install.
On Ubuntu: sudo apt-get install -y nvidia-cuda-toolkit before uv pip install -e ".[gpu]".

OpenAI / Whisper SDK drop-in

from openai import OpenAI

client = OpenAI(
    api_key="not-needed",
    base_url="http://localhost:8002/v1",
)

with open("audio.wav", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="Qwen/Qwen3-ASR-1.7B",
        file=f,
    )
print(transcript.text)  # Clean output — no "language English\n" prefix

Round-trip with qwen3-tts-server

flowchart LR
    A[Text prompt] -->|POST /v1/audio/speech| B[qwen3-tts-server<br/>:8001]
    B -->|MP3/WAV/PCM| C[Audio file]
    C -->|POST /v1/audio/transcriptions| D[qwen3-asr-server<br/>:8002]
    D -->|JSON transcript| E[Recovered text]
    E -.compare.-> A

# 1. Synthesise (qwen3-tts-server)
./test-tts.py "The quick brown fox jumps over the lazy dog." -o sample.wav --format wav

# 2. Transcribe
./test-asr.py sample.wav
# → "The quick brown fox jumps over the lazy dog."

To run both services with a single command:

HF_TOKEN=hf_xxx docker compose up -d

See docker-compose.yml for VRAM budget, startup ordering, and health checks.

Hardware reference (tested)

Primary (benchmarks below)

Component	Spec
GPU	NVIDIA GeForce RTX 4080 SUPER (16 GB VRAM, Ada Lovelace)
CPU	Intel Core i7-14700 KF (20 cores / 28 threads)
RAM	32 GB DDR5
OS	Ubuntu 24.04.4 LTS
Driver	NVIDIA 595.58.03 (CUDA 13.x)

Also validated on

GPU	VRAM	Notes
NVIDIA GeForce RTX 5090 (Vast.ai VM)	32 GB	Driver 580.95.05 / 595.58.03 (CUDA 13.0 / 13.2); three-pass benchmarks — see below
NVIDIA GeForce RTX 4090 (Vast.ai VM)	24 GB	Driver 580.126.09 (CUDA 13.0); container starts and loads correctly
NVIDIA GRID A100D-20C (Vultr vGPU)	20 GB	Driver 550.90.07 (CUDA ≤ 12.4); use `--gpu-memory-utilization 0.55` when co-located with TTS

VRAM budget (16 GB, TTS + ASR on same GPU)

RTX 4080 SUPER:                       16,376 MiB
  TTS (bfloat16 + CUDA graphs):        4,400 MiB  (qwen3-tts-server)
  ASR (fp8 KV + vLLM):                10,400 MiB  (this server, --gpu-mem-util 0.55)
    - Model weights:                   3,870 MiB
    - KV cache (fp8):                  5,130 MiB
    - CUDA graphs + overhead:          1,400 MiB
  Total:                              ~14,800 MiB / ~90%

On an RTX 4090 (24 GB) or RTX 5090 (32 GB) both models fit with headroom to spare; use the defaults (--gpu-memory-utilization 0.65) and let vLLM claim a large KV cache. Measured RTX 5090 co-located footprint: ~28 GB / 32 GB (TTS 4.4 GB + ASR 23.5 GB).

Start order matters: TTS first (fixed footprint), then ASR — vLLM auto-sizes KV cache.

Performance (RTX 4080 SUPER, vLLM + fp8)

Workload	Wall time
5 s mono WAV	~250 ms
30 s mono WAV	~600 ms
Context-primed (`/v1/chat/completions`)	~+50 ms overhead

Performance (GRID A100D-20C vGPU)

Measured on Vultr GRID A100D-20C (20 GB vGPU), driver 550.90.07, CUDA 12.4, container ghcr.io/malaiwah/qwen3-asr-server:latest (cu128 base, vLLM backend), espeak-ng speech at 16 kHz mono — steady-state (warm model, 5 runs averaged).

Workload	Audio duration	Wall time	Real-time factor
Short speech	4.3 s	~243 ms	~17×
Long speech	30.6 s	~1270 ms	~24×

The vGPU hypervisor (driver 550 / CUDA 12.4) adds some overhead compared to bare-metal; the cu128-based container is required — CUDA 13.x containers will fail with Error 803 on driver 550.

Performance (RTX 5090, vLLM + fp8 — three-pass driver comparison)

Measured on Vast.ai NVIDIA GeForce RTX 5090 (32 GB VRAM, Blackwell sm_12.0), Ubuntu 22.04 VM, vLLM 0.14.0 — steady-state (5 runs averaged, first warmup run excluded). ASR WAV files synthesised by qwen3-tts-server on the same machine.

Pass	Driver	CUDA compat	Short (4.2 s)	Short RTF	Long (30.6 s)	Long RTF
1 (stock)	580.95.05	13.0	66 ms	64×	381 ms	80×
2 (upgraded)	595.58.03	13.2	134 ms	32×	822 ms	37×
3 (cache cleared, uncontested recompile)	595.58.03	13.2	67 ms	64×	382 ms	80×

Pass 2 is ~2× slower than Pass 1. The driver upgrade changed vLLM's inductor compile-cache hash (fb51a9fd39 → e78daa734f), forcing a fresh kernel benchmark that selected a slower combination of combo-kernels under concurrent load (TTS competing for GPU). TTS performance was unaffected by the driver upgrade (see qwen3-tts-server benchmarks).

Pass 3 confirms full recovery. After clearing the compile cache and restarting ASR alone (no TTS competition), vLLM recompiled in 3.5 s and selected the same fast kernel combination as Pass 1 — returning to identical steady-state performance.

Practical note: if you upgrade the host driver, delete /root/.cache/vllm/torch_compile_cache/ inside the container and let vLLM recompile uncontested on first startup — this fully recovers Pass 1 speeds (verified: 67 ms / 64× RTF short, 382 ms / 80× RTF long on driver 595.58.03).

Round-trip (TTS → WAV → ASR): 1 380 ms end-to-end for an 8-word sentence.

API reference

`POST /v1/audio/transcriptions` — Whisper-compatible

curl -X POST http://localhost:8002/v1/audio/transcriptions \
  -F model=Qwen/Qwen3-ASR-1.7B \
  -F file=@audio.wav \
  -F language=en          # optional ISO-639-1 hint
  -F response_format=json # json | text | verbose_json | srt | vtt

Response:

{ "text": "Hello from Qwen3." }

The language X\n prefix that Qwen3-ASR internally prepends is always stripped.

`POST /v1/audio/translations` — API compatibility shim

Same interface as /v1/audio/transcriptions. Note: Qwen3-ASR does not translate to English — it transcribes in the detected language. The response includes a _note field explaining this. For actual translation, pass the transcript to a downstream LLM.

`POST /v1/chat/completions` — context-primed transcription

Bias vocabulary, language, or named entities with a system prompt:

{
  "model": "Qwen/Qwen3-ASR-1.7B",
  "messages": [
    {"role": "system", "content": "Speaker uses English/French. May say: Hermes, Honcho, oikos, malaiwah."},
    {"role": "user", "content": [
      {"type": "input_audio", "input_audio": {"data": "<base64>", "format": "wav"}}
    ]}
  ],
  "temperature": 0.0
}

Returns standard chat completion JSON. Requires GPU / vLLM backend.

`GET /health`, `GET /v1/models`

Standard introspection. /health reflects actual vLLM readiness.

`GET /metrics` — Prometheus exposition

Plain-text Prometheus exposition for scraping by Prometheus/VictoriaMetrics/OTEL collectors. Always unauthenticated (standard ops convention). Exposed metrics:

Metric	Type	Labels	Meaning
`qwen3_asr_requests_total`	counter	`method, path, status`	HTTP requests handled
`qwen3_asr_request_duration_seconds`	histogram	`method, path`	Request latency
`qwen3_asr_requests_in_flight`	gauge	—	Requests currently in progress
`qwen3_asr_model_ready`	gauge	—	1 when vLLM is ready, 0 otherwise
`qwen3_asr_backend_info`	gauge	`device, model_id`	Always 1; labels describe the backend

Standard process_* and python_gc_* metrics are included automatically by prometheus_client.

Authentication (optional)

By default the server is unauthenticated (matching typical self-hosted usage). To require a bearer token on /v1/* endpoints set either an env var or a CLI flag:

# Env var (recommended for containers)
docker run -e QWEN_API_KEY=sk-mysecret ghcr.io/malaiwah/qwen3-asr-server:latest …

# Or CLI flag
python server.py --api-key sk-mysecret

Callers must then send Authorization: Bearer sk-mysecret. /health and /metrics stay unauthenticated regardless, so ops scrapers and container health checks keep working.

from openai import OpenAI
client = OpenAI(api_key="sk-mysecret", base_url="http://localhost:8002/v1")

Configuration

Env var	Default	Purpose
`QWEN3_ASR_MODEL_ID`	`Qwen/Qwen3-ASR-1.7B`	Override the HF model
`HF_HOME`	`/root/.cache/huggingface`	Weight cache. Mount a volume here.
`HF_TOKEN`	(unset)	HuggingFace token for gated downloads
`QWEN_API_KEY`	(unset)	Require `Authorization: Bearer <key>` on `/v1/*`

CLI flags (forwarded to vLLM in GPU mode):

Flag	Default	Purpose
`--host` / `--port`	`0.0.0.0` / `8000`	Listener
`--api-key`	(env / off)	Overrides `QWEN_API_KEY` for bearer-token auth
`--gpu-memory-utilization`	`0.9`	Lower to 0.55 when sharing GPU with TTS
`--max-model-len`	`4096`	Context window
`--kv-cache-dtype`	(vLLM default)	`fp8` recommended — halves KV VRAM
`--max-num-seqs`	(vLLM default)	Concurrent requests
`--cpu`	(off)	Force transformers fallback (very slow)

Benchmarking

A standalone benchmark.py measures steady-state wall-clock latency + RTF:

./benchmark.py short.wav long.wav --runs 5
./benchmark.py audio.wav --url http://my-gpu-host:8002 --api-key sk-mysecret

First run per file is warmup (excluded from the average); the rest feed a simple mean ± stdev. The RTX 4080 SUPER / 5090 numbers in the tables above were produced with this script.

Building from source

docker build -t qwen3-asr-server:latest -f Containerfile .

CI builds and pushes to ghcr.io/malaiwah/qwen3-asr-server:latest on every push to main. Version tags (e.g. v0.1.0) additionally produce semver-tagged images (:0.1.0, :0.1).

Supply-chain metadata. Each pushed image carries:

an SPDX SBOM attestation (generated by buildx)
a SLSA build provenance attestation (mode=max)

Inspect either with:

docker buildx imagetools inspect \
  --format '{{ json .SBOM }}' \
  ghcr.io/malaiwah/qwen3-asr-server:latest

Vulnerability scanning. Every push to main / tags runs Trivy against the freshly-built image and uploads SARIF findings to this repo's Security tab. The scan is non-blocking — upstream CUDA base images routinely carry known CVEs outside our control.

Tests

uv pip install -e ".[test]"
pytest -q

Smoke tests run without a GPU or model load.

CPU mode

CPU inference is orders of magnitude slower (multi-second per short clip). Use only for smoke tests, API surface exploration, or environments without a GPU.

For production use, a CUDA GPU with ≥ 6 GB VRAM is required.

Acknowledgements

Qwen team @ Alibaba for Qwen3-ASR-1.7B
vLLM for the inference engine
qwen-asr for the upstream serving CLI

License

MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
tests		tests
.gitignore		.gitignore
Containerfile		Containerfile
LICENSE		LICENSE
README.md		README.md
benchmark.py		benchmark.py
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
server.py		server.py
test-asr.py		test-asr.py

Navigation Menu

Repository files navigation

qwen3-asr-server

What you get

Architecture: proxy, not execv

Host prerequisites (GPU)

Quickstart (Docker / Podman)

Quickstart (uv, no container)

OpenAI / Whisper SDK drop-in

Round-trip with qwen3-tts-server

Hardware reference (tested)

Primary (benchmarks below)

Also validated on

VRAM budget (16 GB, TTS + ASR on same GPU)

Performance (RTX 4080 SUPER, vLLM + fp8)

Performance (GRID A100D-20C vGPU)

Performance (RTX 5090, vLLM + fp8 — three-pass driver comparison)

API reference

`POST /v1/audio/transcriptions` — Whisper-compatible

`POST /v1/audio/translations` — API compatibility shim

`POST /v1/chat/completions` — context-primed transcription

`GET /health`, `GET /v1/models`

`GET /metrics` — Prometheus exposition

Authentication (optional)

Configuration

Benchmarking

Building from source

Tests

CPU mode

Acknowledgements

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

qwen3-asr-server

What you get

Architecture: proxy, not execv

Host prerequisites (GPU)

Quickstart (Docker / Podman)

Quickstart (uv, no container)

OpenAI / Whisper SDK drop-in

Round-trip with qwen3-tts-server

Hardware reference (tested)

Primary (benchmarks below)

Also validated on

VRAM budget (16 GB, TTS + ASR on same GPU)

Performance (RTX 4080 SUPER, vLLM + fp8)

Performance (GRID A100D-20C vGPU)

Performance (RTX 5090, vLLM + fp8 — three-pass driver comparison)

API reference

POST /v1/audio/transcriptions — Whisper-compatible

POST /v1/audio/translations — API compatibility shim

POST /v1/chat/completions — context-primed transcription

GET /health, GET /v1/models

GET /metrics — Prometheus exposition

Authentication (optional)

Configuration

Benchmarking

Building from source

Tests

CPU mode

Acknowledgements

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`POST /v1/audio/transcriptions` — Whisper-compatible

`POST /v1/audio/translations` — API compatibility shim

`POST /v1/chat/completions` — context-primed transcription

`GET /health`, `GET /v1/models`

`GET /metrics` — Prometheus exposition