qwen-tts v0.13.0

Highlights

⚡ Performance — int4 is now an M1 lever (not just x86)

• SDOT-native q4_0 matvec (+ fused QKV): the llama.cpp q4_0×q8_0 trick (quantize the
  activation to int8 once, vdotq_s32 over the nibble blocks) replaces the per-weight
  f32 dequant that used to dominate on M1. int4 now beats int8 on M1 at 4 threads
  (1.7B: int8 RTF 1.84 → int4 1.58, −14%; int4 vs its own legacy path −38%).
• --quant-mixed (int4 Talker + int8 CP) = the fastest 1.7B config on M1 (RTF ~1.53).
• Streaming memory cap: the decoder's pre-transformer KV + latent caches are compacted
  to O(window + chunk) instead of growing with stream length (~29× less for a 10-min
  stream) — bit-exact vs the old unbounded path.
• Prefill via bf16 matmat: a no-convert prefill path, default on non-BLAS builds / a
  lever for weak-BLAS Linux (on M1 Accelerate/AMX still wins, so it stays opt-in there).

🧹 Retired the legacy .vec control-vector system

Superseded by the qlsteer STEER --emotion recipe. Removed: the .vec assets, the
--steer-vector / --steer-weight CLI flags, the .vec builder scripts, and the internal
capture/injection plumbing. Unchanged: --emotion (primaries + Plutchik dyads),
--roughness, and the .qlsteer builder.

⚠️ Breaking: --steer-vector and --steer-weight are gone. They only drove the retired
.vec path; --emotion and --ml-steer are the supported steering surfaces.

✅ Test suite

make test-all is green end-to-end again (three stale test targets — emotion, emotion-ft,
compose — were asserting retired behavior and are updated to the shipped systems).

Full changelog: v0.12.0...v0.13.0

Full Changelog: v0.12.0...v0.13.0

qwen-tts v0.12.0

v0.12.0 — Blended emotions (dyads) + inline emotion switching

A follow-up to v0.11.0 that makes
the emotion system both richer and composable in-text. Still pure C, CPU-default, GPU opt-in.

🎭 Seven blended emotions (dyads)

Emotion steering directions add — summing two primary vectors yields a coherent new emotion. Seven
ear-validated Plutchik dyads now ship as first-class --emotion values (no new capture, no fine-tune):

contempt · awe · nostalgia · disapproval · remorse · outrage · despair

./qwen_tts -d qwen3-tts-1.7b -s ryan -l English --emotion contempt \
    --text "Oh, sure, that's a truly brilliant idea." -o contempt.wav

They use the same steering recipe as the 6 primaries (preset → STEER @ w12). Built by blending the primary
vectors with the new tools/steer/dyad_mix.py.

🔀 Inline emotion switching — many emotions from ONE prompt

Write [emotion] tags inside --text and the engine switches emotion sentence by sentence in a single
generation, clean at the seams, one output file — any primary or dyad, no flags:

./qwen_tts -d qwen3-tts-1.7b -s ryan -l English -T 1.1 --text \
  "[contempt] Oh, sure, that's a brilliant idea. [nostalgia] We used to spend every summer by the sea. [despair] And now there's nothing left." \
  -o switch.wav

[neutral] resets. Works in the HTTP server too (inline markup or the emotion field). Committed audio
samples of the switch happening inside one prompt live in samples/emotion_dyads/.

🧹 One emotion system

The old, weaker .vec CP-steer "mood palette" is retired. Every emotion path — CLI --emotion, inline
per-sentence [emotion], and the server — now routes through the same qlsteer STEER recipe. The server also
clears emotion state per request (no bleed across requests). --roughness and the raw --steer-vector
power-user knob are unchanged.

Notes

Emotion remains 🧪 beta, ear-validated on ryan (English + Italian) for the dyads. The CPU path, primaries,
paralinguistic [tag]s, and GPU backends are unchanged. Full recipe → docs/emotion-THE-recipe.md.

Full changelog: v0.11.0...v0.12.0

Full Changelog: v0.11.0...v0.12.0

qwen-tts v0.11.0

v0.11.0 — Inline paralinguistics, robustness-gated

A focused follow-up to v0.10.0.
The inline paralinguistic [tag] set is expanded and ear-gated on varied natural sentences, plus a
semi-autonomous discovery harness and a make para-demo. Still pure C, zero heavy deps, CPU-default.

🗣️ Inline paralinguistic [tag]s — expanded + gated

Write a friendly tag in --text; the engine performs the event inside the sentence, in one generation,
in the voice's own timbre (never a separate spliced span). No flags — it picks the onomatopoeia anchor +
the right seed per voice for you.

Shipped set ([laugh], [sigh] from before, now joined by):

• [wow] — "wow!" interjection (pairs with --emotion surprise)
• [yawn] — vocal yawn, cross-voice (preset / clone seeds handled for you)
• [scoff] — disdainful tsk (pairs with --emotion disgust)
• [giggle] — sly chuckle, standalone-only (great inline / no-emotion)

./qwen_tts -d qwen3-tts-1.7b -s ryan -l English \
  --text "I opened the box and [wow], it was exactly what I asked for." --emotion surprise -o wow.wav

🚦 Robustness gate (the reason this is a release, not just commits)

Every tag was re-tested by ear on 3 varied natural sentences (English ×2 + Italian) × with/without the
paired emotion, on a preset voice and a cloned voice — because a tag that passes its one validation
carrier can still break on an arbitrary sentence. Outcome:

• Shipped: [wow], [yawn], [scoff] (re-tuned so it no longer stutters on Italian).
• Standalone-only: [giggle] — pairing it with --emotion joy over-drives it into an over-laugh, so
  it's documented as best used on its own.
• Parked: [phew] — clean in Italian but fragile in English (no configuration was robust), so it's held
  back for a future English-robust trigger rather than shipped half-working.

Honest insight: --emotion joy is the aggressive one for paralinguistic events — joy-paired tags
tend to over-drive on long English sentences, while disgust/surprise/sad compose cleanly.

🛠️ Tooling

• make para-demo — renders the shipped tag set on natural sentences with the exact commands + afplay
  links, so you can hear each one.
• E1 discovery harness (tools/para/) — a semi-autonomous hunter: sweep runner + a CNN14 + CLAP screener
  that shortlists candidate onomatopoeiae, with the ear as the final judge. Offline Python tooling only
  (never touches the C engine).

Notes

Paralinguistics remains 🧪 alpha — hit-or-miss across some voices/languages (laughs land best). The
CPU inference path, emotion system, and GPU backends from v0.10.0 are unchanged. Full WIN/KO trail lives in
docs/para-experiments.md.

Full changelog: v0.10.0...v0.11.0

Full Changelog: v0.10.0...v0.11.0

qwen-tts v0.10.0

v0.10.0 — Emotion, expressivity & GPU backends

The two biggest additions since v0.9.0 (243 commits): a complete emotional-expressivity system
and three GPU backends (Apple Metal + NVIDIA CUDA) with server request-batching. Still pure C,
zero heavy deps, CPU-default — GPU is opt-in.

🎭 Emotion & expressivity — solved

• --emotion <sad|joy|anger|fear|disgust|surprise> — one flag, works on preset AND cloned voices.
  THE recipe: preset voices → pure steering (ryan_<emo> @ w12, clean in every language); cloned voices
  → COMBINE (language .expr fine-tune + steering + instruct). Native preset per language (JA/KO/ZH/…).
• Mixed top-win system: hard-won steering vectors + CSP fine-tune (trained on the Emozionalmente
  dataset) applied together — better than either alone. Tiny steer vectors ship in presets/steer/emotion/.
• --instruct strength/speed control (free-form: strong/very-strong, plain-English speed).
• Per-sentence dynamic emotion in the server — inline [joy]…[sad]… markup switches mood paragraph by
  paragraph (/v1/tts + streaming), plus --emotion/volume/rate JSON fields.

🗣️ Paralinguistics — inline [tag]

• [laugh] / [sigh] inline tags — the event is rendered in the voice's own timbre, in one
  generation (no splice/seam). Universal mapping validated across voices & languages.

🖥️ GPU backends (opt-in) — Apple Metal & NVIDIA CUDA

Whole fused pipeline resident on the GPU (weights + KV + activations on device, one command buffer/step).

• Apple Metal (make metal): resident fused Talker + device-frame Code Predictor. M1 0.6B RTF 1.5→0.60;
  M2 Pro 0.6B 0.36–0.39 / 1.7B 0.48–0.53, streaming TTFA 314 ms. Bit-identical to the CPU decoder.
• NVIDIA CUDA (make cuda): resident fused + cuBLAS convs + CUDA graphs. 1.7B RTF 0.44 on a
  ~270 GB/s GPU (RTX 4060-class); scales with bandwidth (→ ~0.12 on a 4090-class).
• Server request-batching (--serve --batch-size N) on all three backends — a throughput/parallelism
  lever (serves N concurrent users in ~the time of one). CUDA ~3.35× (B=8), Metal ~2.8× (B=4); batch output
  bit-identical to single-stream. First cross-backend RTF matrix measured on real silicon (Apple + NVIDIA).

🔧 Also

• Cross-hardware bench tooling (bootstrap_m2.sh one-command rental setup, make bench-matrix), and a
  documented GPU-performance section in the README (single-stream latency vs batching throughput).
• Correctness held throughout: golden mel-corr ≥0.99 on 0.6B & 1.7B; GPU paths gated by argmax self-tests.

Full details: docs/emotion-THE-recipe.md · docs/cuda-performance.md · docs/hardware-testing.md

---

Builds clean (make blas / make metal / make cuda); Code Quality (clang-tidy) + CodeQL + Memory-Safety CI green.

Full Changelog: v0.9.0...v0.10.0

qwen-tts v0.9.0

v0.9.0 — x86 SIMD: AVX2 + AVX-512/VNNI + cross-OS threading

The engine is no longer Apple-only. The hot-path matvec/attention kernels now have full x86 SIMD with a runtime ISA guard and a scalar fallback.

Highlights

• AVX2 + FMA twins for every hot kernel (matvec bf16/int8/q4_0, argmax, all 3 attention variants).
• AVX-512: __m512 16-wide bf16 matvec (SIMD=avx512) + AVX-512-VNNI native int8 dot _mm512_dpbusd_epi32 (SIMD=avx512vnni).
• Cross-OS decode threading: GCD on macOS + a persistent pthread pool on Linux/Windows (was macOS/GCD-only).
• INT8 now includes the 0.6B Talker; INT4 (Q4_0) on Talker + Code Predictor — the memory-bound x86 lever.
• Concurrent HTTP server: --workers N (request-level parallelism; per-worker context clones share the read-only weights).
• New tooling: ./qwen_tts --caps (report compiled SIMD/threads), ./qwen_tts --self-test (cross-ISA kernel numeric correctness gate), and tests/x86_bench.sh (per-box RTF A/B).

Validated

• Apple M1 (NEON+SDOT) · Ryzen 7 6800H (AVX2) · AMD EPYC 9555P / Zen5 "Turin" (AVX-512+VNNI).
• Full cross-device RTF table in docs/performance.md.

Build

make blas                    # x86 default: portable AVX2+FMA
make blas SIMD=avx512vnni    # AVX-512 + VNNI (Zen4+/Intel)
make blas SIMD=scalar        # portable fallback

Note: single-stream RTF is memory/cache-bound (the Code Predictor re-reads its weights 16x per frame) — quantization (int8/int4) and a cache-rich CPU matter more than SIMD width. Many-core servers are best for throughput.

Full Changelog: v0.8.2...v0.9.0

qwen-tts v0.8.2

Bug Fix: --instruct Now Actually Changes Prosody

What was wrong (fixes #16)

The sampling defaults for the talker and the code predictor (sub-talker) were lower than those used by the reference Python implementation. This suppressed the effect of --instruct prompts (angry, sad, whispering, …) to the point where the generated audio was nearly indistinguishable from the baseline.

What changed

Aligned the hard defaults in qwen_tts_load() and main.c with the hard_defaults in qwen_tts/inference/qwen3_tts_model.py from QwenLM/Qwen3-TTS:

Parameter	Before	After (official)
Talker temperature	0.5	0.9
Code predictor temperature	0.0 (greedy)	0.9 (sampled)
Code predictor top_k	1	50
top_p, rep_penalty, top_k (talker)	1.0 / 1.05 / 50	unchanged

These defaults are used unconditionally by the official generate_custom_voice / generate_voice_clone / generate_voice_design paths — i.e. they are not conditional on whether --instruct is provided.

A/B test — attached WAV samples

Same text, same seed (--seed 42), same speaker (-s ryan), Qwen3-TTS-12Hz-1.7B-CustomVoice.

English — "I cannot believe you did that to me." + --instruct "Speak in a very angry and aggressive tone":

• issue16_en_angry_OLD_t0.5.wav — v0.8.1 defaults, sounds close to neutral baseline
• issue16_en_angry_NEW_t0.9.wav — v0.8.2 defaults, noticeably more intense / angrier delivery

Italian — "Non posso credere che tu mi abbia fatto questo." + --instruct "Parla con tono estremamente arrabbiato e aggressivo, urlando":

• issue16_it_angry_OLD_t0.5.wav — v0.8.1 defaults, essentially the same prosody as the no-instruct run
• issue16_it_angry_NEW_t0.9.wav — v0.8.2 defaults, sharper, more aggressive

Notes

• --temperature / -T on the CLI can still override the default.
• Non-instruct generation on a preset speaker is mostly unchanged on short English prompts and gets somewhat more prosodic variation on longer Italian prompts (expected side-effect of higher sampling temperature; matches the behavior of the reference implementation).
• This is a known upstream limitation of the instruct feature, especially on cloned voices — see QwenLM/Qwen3-TTS discussion #231. These aligned defaults make the effect as strong as it is in the reference, but do not magically unlock stronger emotion control on top of what the model is capable of.

Full Changelog: v0.8.1...v0.8.2

Full Changelog: v0.8.1...v0.8.2

qwen-tts v0.8.1

What's New

AVX2 Parity for RMSNorm Kernels

Added AVX2 paths (16-wide, FMA) for the three most-called norm functions:

• qwen_rms_norm — every transformer layer
• qwen_rms_norm_per_head — Q/K head normalization
• qwen_rms_norm_residual — fused residual+norm

On x86 servers these were falling back to scalar. Now use _mm256_fmadd_ps for sum-of-squares and 16-wide multiply-normalize loops.

Phase 18 Experiments Documented

Tested and documented several approaches that didn't improve performance on Apple M1:

• pthread thread pool: 8% slower than GCD (Apple's kernel-optimized dispatch)
• 4-row fused matvec: 7% slower (register spill at 16 NEON accumulators)
• Threading threshold tuning: 3% slower (1024-row matrices still benefit from 4 threads)
• CP↔Talker pipeline overlap: infeasible (strict serial dependency chain)

Bug Fix

Fixed INT8 CP status message showing [INT8] even when quantization was skipped.

Full Changelog: v0.8.0...v0.8.1

Full Changelog: v0.8.0...v0.8.1

qwen-tts v0.8.0

What's New

Fused Residual-Add + RMSNorm (-21% RTF on short text)

New qwen_rms_norm_residual() kernel fuses x += residual with RMSNorm in a single pass, eliminating redundant vector reads. Applied to both Talker (28 layers) and Code Predictor (5 layers × 17 passes/frame).

INT8 Load Bug Fix (critical)

--int8 flag was set after qwen_tts_load() returned — the Talker and CP loaders never saw it. INT8 quantization was silently skipped on all models since it was first implemented. Now fixed with qwen_tts_load_ex().

Also: INT8 on 0.6B (hidden=1024) causes generation hang — now auto-skipped. INT8 only activates on 1.7B+ where matrices are large enough to benefit.

Software Prefetch in BF16 Matvec

Prefetches weight rows 4 ahead in the fused 2-row matvec kernel. Marginal (~1-2%) on M1 but cross-platform and zero-cost.

Performance (Apple M1 8-core, 16 GB)

Test	v0.7.0	v0.8.0	Improvement
0.6B short normal	1.69	1.34	-21%
0.6B long normal	1.29	1.35	stable
0.6B stream	1.31-1.33	1.32	stable
1.7B long normal	1.97	1.99	stable

Phase 18 Roadmap

Added comprehensive performance roadmap in PLAN.md targeting RTF ≤ 1.0, including profiling data showing Code Predictor is 67% of per-frame time with 30% GCD dispatch overhead.

Full Changelog: v0.7.0...v0.8.0

Full Changelog: v0.7.0...v0.8.0

qwen-tts v0.7.0

What's New

Vectorized BF16→F32 Codec Embedding Lookups

New qwen_bf16_to_f32_vec() SIMD kernel (NEON 8-wide + AVX2 8-wide) replaces all scalar bf16→f32 conversion loops in the hot path:

• lookup_codec_embed() in the main generation loop
• Code Predictor code0 embedding lookup (talker codec_embedding)
• Code Predictor codebook 1–15 embedding lookups (14× per frame)

Zero extra memory — conversions happen in-place to stack/existing buffers. Avoids the F32 pre-conversion approach which caused +120 MB mmap cache pressure on the 0.6B model.

Performance Improvements (Apple M1 8-core, 16 GB)

Metric	Before (v0.6.0)	After (v0.7.0)	Improvement
1.7B long normal RTF	2.10	1.97	~6% faster
1.7B short stream RTF	3.67	2.59	~30% faster
0.6B (all configs)	1.29–1.71	1.29–1.69	stable

The vectorized kernels benefit the 1.7B model most due to larger embedding dimensions (2048 vs 1024).

Full Changelog

• 667e5d1 Vectorize bf16→f32 conversions in codec embedding lookups
• 237c185 Update RTF benchmarks with new numbers
• Updated docs/performance.md optimization history

Full Changelog: v0.6.0...v0.7.0

Full Changelog: v0.6.0...v0.7.0

qwen-tts v0.6.0

What's New

SIMD Element-wise Ops

• qwen_add_inplace, qwen_mul_inplace, qwen_vec_scale_inplace now use NEON (8-wide, 2x unrolled) and AVX2 (8-wide) with scalar tail — called per-layer in residual paths

Allocation & Memory Cleanup

• Removed dead dec_up / cp_dec_up NULL pointers and free() calls
• Codec PAD/BOS embeddings and speaker ref_norm cached at load time (zero alloc per generation)
• Sampling work buffers use aligned_malloc for SIMD-friendly access
• KV cache initial allocation reduced from 2048 → 512 slots (~12MB saved on 0.6B)
• RoPE cache sized to actual max_tokens instead of hardcoded 8192
• Prefill buffer realloc uses doubling strategy (fewer reallocs in server mode)

Smart Thread Default

• Default threads now max(4, ncpus/2) — stays at 4 on M1 (same as before), scales up on larger machines

Benchmark Suite

• New make bench (quick) and make bench-full (comprehensive) targets
• Tests short + long text, normal + stream, server cold/warm, instruct, INT8, .qvoice
• Auto-skips missing models and voice files — no crashes
• Reports RTF, audio duration, wall time per configuration

RTF Results (Apple M1, 4 threads)

Config	0.6B	1.7B BF16	1.7B INT8
CLI short	1.65–1.71	4.15–4.40	3.69
CLI long	1.29–1.32	2.10–2.11	2.15
Stream short	1.30–1.31	3.67–4.01	—
Stream long	1.30	2.06–2.43	—

Full changelog: v0.5.0...v0.6.0

Full Changelog: v0.5.0...v0.6.0