Our own decoder, YuE2-VAE-RS x3: what the standard one gets wrong, what we changed, and the numbers (the default from 2.0.7)

[igrbible]

YuE2 sings in latents, 64 numbers 25 times a second, and a decoder turns them into 48 kHz stereo. Until now the Studio's default was m-a-p's decoder, which we call Standard. From 2.0.7, the next release, it is our own: YuE2-VAE-RS x3. Here is why, what it is, and what it measures, including where it does worse.

What Standard gets wrong

Standard is a good decoder; to our knowledge nobody had fine-tuned it. Against the originals, though, it leaves two traces of its own structure. We measured them on eight 30-second clips our training never saw.

• A 100 Hz ripple. The envelope of everything above 3 kHz trembles at 100 Hz, which is four times the latent frame rate: the frame grid showing through on sustained tones. It stands +6.5 dB over its neighbourhood on a duduk and +7.2 dB on an orchestra, where the originals sit at +0.6 and +0.2. m-a-p's older Legacy decoder and Blend, the community merge of the two (two thirds Standard, one third Legacy), carry the same ripple.
• Seams at 6 and 12 kHz. These are lines where the upsampling stitches its bands. The 6 kHz line stands +20.6 dB over its local floor on a studio guitar recording, whose original has +5.5 of its own. The 12 kHz line reaches +12.9 dB on speech and +30.0 dB on a shofar's decay, though there far down, at −109 dBFS.

What we built

x3 is Standard's decoder grown three times over, from 66.37 M to 200.76 M stored values. We added residual units of Standard's own form, three at the latent rate and three more in each of its six upsampling blocks. Each new unit began at zero, so before training the grown decoder equalled Standard sample for sample. Then the whole decoder was trained on music against those traces.

The encoder and everything upstream are untouched: x3 reads the same latents as Standard, under any YuE2 backbone, m-a-p's or ours.

Two runs trained for about three and a half days each, on one RTX 3090 apiece. We released two checkpoints:

• x3 v1, from the first run at step 161,953: the end of its steadiest stretch. It saw 262.2 hours of audio in 36.1 GPU-hours.
• x3 v2, from a second branch of the same run with a gentler weight on the >3 kHz envelope, at step 155,030. It saw 252.8 hours of audio in 34.6 GPU-hours.

The numbers

Means over seven music clips against their originals, in dB: excess lower is better, energy loss closer to zero, SI-SDR higher.

	Standard	x3 v1	x3 v2
100 Hz ripple, excess	+2.71	+0.60	+0.36
6 kHz line, excess	+3.59	+0.98	+0.79
12 kHz line, excess	+6.96	+2.16	+2.13
energy lost at 16–20 kHz	−3.19	−1.58	−1.64
SI-SDR	8.54	7.85	7.88
30 s decode on an RTX 3090, s	0.29	0.57	0.56

x3 brings the ripple much nearer the originals: on the duduk from +6.5 to +2.0 (v1) and +0.4 (v2), and on the orchestra from +7.2 to +0.8 and +1.0. On the studio guitar the 6 kHz line falls from +20.6 to +1.8 and +0.3, below the recording's own +5.5: some of its real 6 kHz content goes with the seam.

On 40 generated songs (our benchmark's 20 cards at two seeds), decoded again from their own latents, x3 against Standard:

• the decoder's 25 Hz buzz is lower by 0.96 dB (v1), lower on 31 of the 40 songs;
• stray tones fall from 2.15 to 0.80 a song;
• Whisper's word error moves from 22.15 % to 22.96 %, with a median change of zero;
• the sound is a little brighter, by +0.45 dB per octave of spectral tilt;
• the 6–16 kHz "sand" measure is slightly worse, by +0.28 dB, better on only 10 of the 40.

Where it is worse

• Waveform fidelity is lower: SI-SDR drops on every clip, by 0.3–1.4 dB.
• The floor between harmonics is no better: +1.59 / +1.65 dB over the originals against Standard's +1.22.
• 8–12 kHz is its thinnest place: about 3 dB lost on two clips, where Standard loses 0.6 and 1.6.
• The seam did not vanish: at fine resolution it became a narrow line 1.7–2.2 dB taller than Standard's, inside a notch 4.4–5.0 dB deeper.
• It costs more: twice the decode time, three times the weights (0.8 GB), and 60 latent frames of tile context on each side instead of 12.

Hear it yourself

Our page plays those 40 songs through Standard, x3 v1 and x3 v2, from the same latents and at the same loudness; switch while one plays and it goes on from the same second. The numbers say where to listen: on these generated songs the 100 Hz ripple is small in Standard already (+0.02 dB on average), so the differences measured there are the 25 Hz buzz, the stray tones, the slightly brighter tilt, and the seams at 6 and 12 kHz. Which decoder sounds better is for your ear to judge.

In the Studio

From 2.0.7, RS x3 v1 is the default decoder and comes with the install and the update. It runs in float32 only, since half precision would put its error where its gains are; on a card short of memory the Studio unloads the backbone before the decode instead.

• The decoder for the songs to come: the VAE list in the ☰ menu, and the same list in Engine → Hardware → Compute.
• Which decoders are offered: Engine → Defaults → Decoders offered (up to three). Ours, RS x3 v1 and RS x3 v2, are offered by default. Tick Standard to have m-a-p's decoder beside them.
• Another decoder for a take you already have: press + RS x3 v1 (or another decoder) on the take's card. A take that kept its latents decodes again in seconds, and both versions sit side by side.

Two documents

• The study (its HTML is also in the repo): the method, every hour of training, the curves, and every measurement, with Legacy and the Blend as references.
• The integration guide, for programmers: PyTorch, yue2_infer, ComfyUI, ggml engines, and verify.py.

The weights (safetensors and F32 GGUF) are in VAE-RS/ of our fine-tunes' repository.

Licence and thanks

The weights are under CC BY-NC 4.0, the licence of m-a-p's YuE2-Vae they are adapted from; the code in VAE-RS/kit/ is under Apache-2.0 with its NOTICE. Thank you, m-a-p, for YuE2, its VAE and its open weights: every number here begins from your decoder.

Viktor Zhuromskyy and Claude (Anthropic), Ruach Studio