Skip to content

Qwen3-TTS Native v0.3.0

Choose a tag to compare

View all tags
@luka-loehr luka-loehr released this 18 Jul 13:37
· 11 commits to main since this release
v0.3.0
c193e25

Qwen3-TTS Native v0.3.0

A performance-focused release: the native engine now performs cross-request
lockstep batched decoding, captures the complete decode frame as a CUDA graph,
and offers an opt-in INT8 weight-only decode mode.

Immutable deployment

ghcr.io/luka-loehr/qwen3-tts-native@sha256:276990b3b693136d6bd93b098682f3860751afb3f5baa99eb5aec2b85c086575
  • Source commit: c193e259cac16219d18d312755e4bbeef82f04f0
  • Annotated source tag: v0.3.0
  • OCI platform: linux/arm64
  • Model: Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign
  • Model revision: 5ecdb67327fd37bb2e042aab12ff7391903235d3

Deploy only the complete digest reference above.

What changed

  • Lockstep batched decode (talker ABI v3). When several requests generate
    concurrently, one batched native call performs the talker and 15-step
    code-predictor decode for every active session, so each shared weight matrix
    is read from memory once per codec frame instead of once per session. KV
    caches, sampling state, histories, random states, and delivery leases stay
    request-local.
  • CUDA-graph frames. The complete batched predictor+talker frame is
    captured once per session tuple and replayed thereafter, eliminating
    per-frame launch overhead for the ~1,600-kernel decode sequence.
  • Opt-in INT8 decode (QWEN3_TTS_INT8_DECODE=1): decode GEMM weights are
    quantized to per-output-channel symmetric INT8 at engine load, directly from
    the pinned BF16 artifact; the checkpoint identity and artifact contract are
    unchanged, and prefill stays BF16. The default engine remains BF16.
  • First-frame priority: a session's first frame bypasses the lockstep
    rendezvous, so first-audio latency does not pay a batching penalty.

Measured performance (idle DGX Spark, all 24 workload languages natural-EOS)

Metric v0.2.0 engine v0.3.0 BF16 v0.3.0 INT8
Single-stream RTF 0.76 0.72 0.47
Single-stream TTFA p50 81 ms 75 ms 60 ms
B3 aggregate RTF / per-request 0.59 / 1.65 0.41 / 1.10 0.33 / 0.90
B6 aggregate RTF / per-request 0.59 / 3.22 0.36 / 1.91 0.31 / 1.69

These figures come from short engineering runs with the repository's external
HTTP client on an otherwise idle Spark. They are engine-comparison numbers,
not a replacement for the full schema-v1.2 controlled evidence bundle; the
published controlled benchmark and research paper remain those of the
v0.1.0 release.

INT8 mode changes decode numerics (per-channel quantization error is on the
order of 4e-3 relative); all transport, EOS, and language gates pass in both
modes. Listening review of INT8 output is the deployer's responsibility.

Qualification performed for this release

  • cargo fmt, clippy -D warnings, and tests for all six crates at the
    release commit.
  • In-build model identity gates (manifest and weight SHA-256, full native
    artifact validation) and release-metadata license/SBOM validation.
  • Container qualification on DGX Spark: hardened run profile (read-only root,
    cap-drop ALL, no-new-privileges), readiness within 20 s, all 24 workload
    languages with natural EOS in BF16 and INT8 modes, and the B1/B3/B6
    benchmark table above.
  • Not performed for this release: the full 12-run schema-v1.2 controlled
    comparison, SoX audio validation, and human listening review.
Assets 2
Loading