Add support for Yue2

[Lastofthefirst]

A new music generation model was released today:

https://map-yue2.github.io/

[0xShug0]

ah checked a few days ago and the last update was a year ago. I will take this.

[tianshuo]

The demos sounds amazing, esp. Suno V6 is restricting.

[WilliamLiao2015]

Add the GitHub repo link of YuE2 here in case anyone doesn't find it: https://github.com/multimodal-art-projection/YuE

[0xShug0]

Call it a day and continue optimizing tomorrow. Now RTF is cpp 0.3 vs python 0.31. ~14G VRAM. using the official example https://huggingface.co/m-a-p/YuE2-3B/blob/main/examples/tonight-awake.json.

Update: in dev now.

yue2_demo.mp4

cpp_current_128k.mp3

[0xShug0]

I think Q4_0 is safe. Added Q4_0 demos to the HF repo. Current perf:

Combo	Audio dur	Wall	RTF	Peak VRAM
BF16 main + F32 VAE	224.96s	60.46s	0.2688	12535 MiB
Q8_0 main + F16 VAE	194.84s	38.81s	0.1992	8867 MiB
Q4_0 main + F16 VAE	221.12s	44.15s	0.1997	7755 MiB

[0xShug0]

@IIIIIllllIIIIIlllll Let me know if it can run on AMD :). This model is pretty impressive.

Check this out: heavy metal jingle bells
Jingle_bells.wav

[IIIIIllllIIIIIlllll]

@IIIIIllllIIIIIlllll Let me know if it can run on AMD :). This model is pretty impressive.

Check this out: heavy metal jingle bells Jingle_bells.wav

no problem ;)

[prookyon]

Do you also plan to add support for https://huggingface.co/m-a-p/SheetSage2 - their model for creating ABC sheets to support cover functionality?

[overallbit]

Call it a day and continue optimizing tomorrow. Now RTF is cpp 0.3 vs python 0.31. ~14G VRAM. using the official example https://huggingface.co/m-a-p/YuE2-3B/blob/main/examples/tonight-awake.json.

Update: in dev now.
yue2_demo.mp4

cpp_current_128k.mp3

is this on windows? can you upload it?

[0xShug0]

@tianshuo @prookyon Yue2 on dev already supports ABC input: you can pass an existing score.abc with abc_file or inline abc when using cot=melody / cot=full. ABC export/edit/re-import is more of a UI-side workflow and hasn’t been implemented yet. Happy to support this once the main model impl is stabilized.

SheetSage 2 inference has been implemented https://github.com/0xShug0/audio.cpp/blob/dev/src/framework/midi/sheetsage2_runtime.cpp. It will be exposed as a standalone model later.

[0xShug0]

@overallbit Building now! Once finished, you can find them in the “Artifacts” section on this page: https://github.com/0xShug0/audio.cpp/actions/runs/34642627647 (default step is 32. 8 is good enough)

[zestybaby]

Love this one so far using Windows, but if I use ABC score feature to generate a song, my system drive's remaining space would shrunk to less than 200 MB from 8 gigs left in usual.

Also, can you adapt the Cover feature from YuE 2's original WebUI into this project with SheetSage2 and MERT2?

[IIIIIllllIIIIIlllll]

@IIIIIllllIIIIIlllll Let me know if it can run on AMD :). This model is pretty impressive.

Check this out: heavy metal jingle bells Jingle_bells.wav

The same prompt: https://huggingface.co/m-a-p/YuE2-3B/blob/main/examples/tonight-awake.json

BF16 weight + F32 vae

RTF: ~0.5

It's so big I can't upload. Are there any other prompts I can test?

[0xShug0]

@IIIIIllllIIIIIlllll Nice! You can convert it to 128k mp3. Or use one of the demo prompts from https://map-yue2.github.io/

[0xShug0]

Love this one so far using Windows, but if I use ABC score feature to generate a song, my system drive's remaining space would shrunk to less than 200 MB from 8 gigs left in usual.

Also, can you adapt the Cover feature from YuE 2's original WebUI into this project with SheetSage2 and MERT2?

SheetSage2 has been added in the dev as a model runtime. I will look at UI carefully latter (not a UI person, and already introduced regression in UI in my commit yesterday)

[0xShug0]

And how can I get an ABC generated on request before a song is also generated on these notes?

@napa3um check the video here #499 (comment) if you open the ABC section you can provide the the audio and extract and edit them.

[napa3um]

And how can I get an ABC generated on request before a song is also generated on these notes?

@napa3um check the video here #499 (comment) if you open the ABC section you can provide the the audio and extract and edit them.

https://github.com/multimodal-art-projection/YuE/blob/main/docs/generation.md#save-and-inspect-a-plan-before-synthesis - As I understand it, YuE2 has the ability to generate an ABC plan without generating the final result.

[0xShug0]

@napa3um We do have the ABC export feature, but it currently requires running the full generation pipeline once. It exports the ABC token IDs as readable ABC text rather than as raw integer token IDs.
https://github.com/0xShug0/audio.cpp/blob/main/docs/models/yue2.md#generated-abc-export

./build/debug/bin/audiocpp_cli \
  --task gen \
  --family yue2 \
  --model models/Yue2-3B-GGUF \
  --backend cuda \
  --threads 8 \
  --lyrics "..." \
  --request-option style="English, folk pop" \
  --request-option cot=full \
  --seed 1234 \
  --out yue2.wav \
  --out-dir yue2_out \
  --log
# -> yue2_out/score.abc

[0xShug0]

@napa3um You should now be able to skip generating the full song and just export the ABC or semantic tokens. Kudos to @engival!
The cmd: https://github.com/0xShug0/audio.cpp/blob/main/docs/models/yue2.md#stopping-early

[nazeshinjite]

YuE2 on Apple Silicon: M5 Max 128 GB, --backend metal, main 61e062d, YuE2-only build with AUDIOCPP_DEPLOYMENT_BUILD=ON. Works out of the box, no patches.

Case	main / VAE	steps	cot	audio	wall	RTF	peak mem
tonight-awake	q8_0 / f16	8	full	216.3 s	115.2 s	0.53	9.8 GiB
tonight-awake	bf16 / f32	8	full	204.1 s	116.4 s	0.57	11.5 GiB
tonight-awake	q8_0 / f32	32	full	216.3 s	233.0 s	1.08	9.8 GiB
demo1	q8_0 / f16	8	off	40.6 s	13.7 s	0.34	10.5 GiB
demo2	q8_0 / f16	8	full	39.8 s	13.5 s	0.34	6.0 GiB

Wall is session.wall_ms (model load excluded).

• Flash attention on AR and NAR; the full song runs as one NAR chunk (355 MB graph buffer); CPU time is negligible.
• Reruns are byte-identical.
• cot=off, melody and full plus external abc_file all work. Stem-ASR lyric match was 91–100%, the same range as your CUDA example renders.

Two questions:

1. ar_runtime.cpp skips ggml_round_bf16 for the prefix K/V on Metal (3abd774, Sep 14), and the vendored ggml gained a Metal ROUND_BF16 kernel on Sep 19 (breeze: enable bf16 activation rounding on Metal #554). I tried removing the skip (skip_bf16_round = false): it runs on the GPU with no change in CPU or wall time, the semantic stream is byte-identical, and output stays deterministic. The render moves slightly: null vs unpatched is −52 dB on 40 s clips (q8_0 and bf16) and −41 dB on tonight-awake. I don't have the Python reference dumps, so I can't tell whether that's closer to the reference; yue2_nar_parity_probe --backend metal with and without the skip would settle it. Is the skip still needed, or is it left over from before that kernel?

2. You mentioned above that the dev-release CLI/server default is 32 steps, but on main 61e062d it's 8 (types.h:58, session.cpp:187, docs/models/yue2.md), and the sidecar's ode_steps: 32 isn't read. Is 8 intended for main?

[0xShug0]

@nazeshinjite Thanks for testing! For your two questions: (1) That patch may no longer be needed given the recent ggml Metal changes. (2) Yes. 8 is intended for main for a quick smoke run. Once it’s merged into main, the dev release will become stale and won’t be updated unless there's something new to test. But I recent found a better default could be cfg = 1.0 and 32 NAR steps.

[incf-ai]

I got it running, compared to the original python from the model authors I almost feel like this implementation struggles more with instrumentals and getting the length correct (mostly unwanted vocals, but maybe I just lucky with random seeds and settings with the python). All of this is based off of a low sample size doing covers with sheetsage.

It's almost as if sheetsage2 with the original python captured layered instruments better. (edit: probably not this)

Also, some of my instrumental gens, like the attached were with totally empty lyrics (no tags whatsoever), but the web UI forbids that.

edit: I tired using the same seed, abc file, caption, parameters, etc. as the python. It's hard to be say anything objective about it since it's expected to be different. But to my ear the 2x audiocpp gens I tried are missing a lot of information that the python version successfully reproduced.

instrumental cover generated with original python
audio insturmental python.mp3

[0xShug0]

@incf-ai Did you test Python and audio.cpp with the same weight dtype? Could you please share the cmd, seed, abc file etc you used?
I’d appreciate more details here. In a controlled setting, audio.cpp and the Python reference implementation are parity-safe for YuE2 and SheetSage2, which confirms the correctness of the math and logic. If the issue is caused by precision drift, the only real solution is to force higher precision at the cost of increased VRAM usage.

Also, you mentioned a “missing score” issue with SheetSage2 but never followed up with details. I haven’t been able to reproduce it, so I’d appreciate more information on that as well.

If you are using https://huggingface.co/Mothersuperior/YuE2-instrumental-cot-full-loras, the prompt needs to be at least "[instrumental]".

Update: BTW you can check https://github.com/0xShug0/audio.cpp/tree/main/tests/yue2 these are the parity test scripts. One thing we can’t fully control is the randomness in generation, but you can pass noise or semantic tokens generated by the Python implementation into audio.cpp

[incf-ai]

Let me know if I missed anything or can provide anything else, I'd be happy to answer any questions.

Did you test Python and audio.cpp with the same weight dtype?

yes

Could you please share the cmd, seed, abc file etc you used?

see json in zip file below, and screenshot below

Also, you mentioned a “missing score” issue

The missing score appears to have been an issue on my end with not specifying the correct vulkan device number and it possibly erroring out when loading sheetsage but not the main music model

If you are using

No Lora was used with either the python or audio.cpp

--

Using the following to inspect the safetensor files

python3 -c "
import json, struct, sys, collections
with open(sys.argv[1], 'rb') as f:
    n = struct.unpack('<Q', f.read(8))[0]
    h = json.loads(f.read(n))
h.pop('__metadata__', None)
for k, v in h.items(): print(k, v['dtype'], v['shape'])
print(collections.Counter(v['dtype'] for v in h.values()))
" model.safetensors

audiocpp vae: FP32
audiocpp model: BF16
python vae: FP32
python model: BF16
python sheetsage2: FP32
python mert2: FP32

Python input/output folders - include json describing the exact parameters
cyclemagic-transcription-song-cover.zip

input audio also available upon request

screenshot of audiocpp page. (I tried cfg=1 and cfg=2)

[0xShug0]

@incf-ai Thanks! These give us something to work from. Could you also share the input audio?

[incf-ai]

python input: irdd-faexoptim-cyclemagic.mp3
python output: audio.mp3

[0xShug0]

I'll see if I can find any issues and fix them in time for tomorrow’s release. Let’s get YuE2 as close to perfect as we can!

[0xShug0]

@incf-ai I removed the empty-string check and used the Python parameters to generate this WAV (converted to 128 kbps for sharing). The empty check is incorrect and shouldn't be enforced, otherwise vocals can leak through. I will fix this.

audio-831001-128k.mp3

Your logs show that Python used PyTorch 2.8.0 with ROCm 6.4 on an AMD GPU, while audio.cpp used Vulkan. This can change both numerical results and the random sampling trajectory, so the same seed does not guarantee identical output across the two backends.

And if you replay the same semantic tokens and noise in C++, the C++ and Python outputs are highly similar (CUDA):

alignment drift: 0 samples throughout
Waveform correlation: 0.999177
RMSE: 0.00544
Local cosine similarity (random segments of audio): 0.9911–0.9999

Check out the outpus:

cpp_controlled.mp3
python_controlled.mp3

[incf-ai]

They do sound similar and the similarity scores your present are high. Seems good.

I hadn't even thought to control for ROCM vs Vulkan.

I had halfway expected random seeds to not be reproducible due to the complexity and differences between the python and cpp engines.

[0xShug0]

Moved to Discussion. In case you’re interested, this branch has additional VRAM optimizations that let YuE2 fit in just 1.7 GB of memory on an iPhone.

Yue2_ios_demo_20x.MP4