sd: sync to master-646-0baf721

[wbruna]

On top of #2213 . LTX 2.3 support, some sampler changes, and a vocab file refactor.

Several added files:

otherarch/sdcpp/guidance.cpp
otherarch/sdcpp/guidance.h
otherarch/sdcpp/ltx_audio_vae.h
otherarch/sdcpp/ltx_vae.hpp
otherarch/sdcpp/tokenizers/gemma_tokenizer.cpp
otherarch/sdcpp/tokenizers/gemma_tokenizer.h
otherarch/sdcpp/tokenizers/vocab/clip_merges.hpp
otherarch/sdcpp/tokenizers/vocab/gemma_merges.hpp
otherarch/sdcpp/tokenizers/vocab/gemma_vocab.hpp
otherarch/sdcpp/tokenizers/vocab/mistral_merges.hpp
otherarch/sdcpp/tokenizers/vocab/mistral_vocab.hpp
otherarch/sdcpp/tokenizers/vocab/qwen_merges.hpp
otherarch/sdcpp/tokenizers/vocab/t5.hpp

and a few removed:

otherarch/sdcpp/tokenizers/vocab/clip_t5.hpp
otherarch/sdcpp/tokenizers/vocab/mistral.hpp
otherarch/sdcpp/tokenizers/vocab/qwen.hpp

so I ended up sorting and rebuilding the source file list.

To fix the linking errors, I've added load_gemma_merges and load_gemma_vocab_json functions with search+replace from the mistral ones.

Edit: updated to master-633-5b0267e to get a few new fixes.

Edit 2: updated to master-637-ef92a00 for more LTX stuff, and a LoRA offloading performance fix.

Edit 3: updated to master-642-3a8788c, with more LTX stuff, and a small refactor.

Edit 4: updated to master-646-0baf721 for more LTX changes.

[LostRuins]

Alright now that the other PR is merged, this is next on the list after I test out the gemma vocab extraction and minifications

[henk717]

Does this already enable LTX or will that require additionals?

[wbruna]

It's just the sync for now. It'll still need at least the gemma external merges and embed files, support for two additional models (embeddings connectors and audio VAE), and likely a file remapping for the llm.

[wbruna]

@LostRuins , I'm hitting a crash bug on the current concedo_experimental version related to offloading with LoRAs; I believe the LoRA caching code is interacting badly with the recent backend changes. But apparently the "LoRA offloading performance fix" from this PR fixes it, too.

Do you intend to merge this soon? I can make a separate PR for the fix if you don't.

[LostRuins]

I was planning to defer this PR until I could look at LTX 2.3, but I can merge it early if you prefer. Would that be easier?

[wbruna]

I was planning to defer this PR until I could look at LTX 2.3, but I can merge it early if you prefer. Would that be easier?

It would be a bit easier to test, since we'd be in sync with upstream. There is a risk of regressions, but I believe it would be better for us to find them sooner anyway.

[LostRuins]

alright merged. gonna do some quick tests for regressions then look into ltx

[LostRuins]

Okay z-image is completely broken now (won't load). perhaps i was too hasty to merge lol

Tested:
SD1.5 - ok
SDXL - ok
Z-Image - cannot load
Qwen Image Edit - severe degraded output

Edit: Now it seems to work again. very weird. but i do have the images from when it failed.

[LostRuins]

Now that I've done some testing I think we have regressions even before this. Qwen Image Edit is no longer loading correctly with a LoRA even before this PR. It works fine if no LoRA was selected. let me try to find out where.

Edit: seems like it works after I did a full revert + full merge again. No idea why - could be vulkan shader caching, some merge squash failure or something else. very weird...

[LostRuins]

Anyway I believe it's all good now.

I also slapped on the hidream patch you suggested in d774184

Just a hacky solution to make hidream work for now until if/when leejet reviews it in leejet/stable-diffusion.cpp#1496

[wbruna]

Oof, alright; I'm glad it's working now. Rebuilding everything to test here.

I also slapped on the hidream patch you suggested in d774184

Just a hacky solution to make hidream work for now until if/when leejet reviews it in leejet/stable-diffusion.cpp#1496

Alright. Yeah, it likely needs a precision bump for Vulkan somewhere.

[LostRuins]

Does it work fine for you?

[wbruna]

Working fine, both on Vulkan and on ROCm. Didn't even need a full rebuild.