Is training vae mandatory?

[XDM-Inc]

So I finally figured out how to set up a working version of simple tuner on Linux for Radeon's but compared to other trainers is very slow about two times slower And way more susceptible to out of memory errors. I suspect it's something to do with vae training, which is something I never do in other software, but I can't find a way to turn it off here. Is there a way to turn it off and not train the vae?

[bghira]

it's not training the VAE - the pre-cache ensures it is unloaded before training runs. but ROCm is pretty tricky to get right, and isn't something i can really debug for users. I removed my AMD GPU - bugs will go uncaught more and more over time.

[XDM-Inc]

If that's so, wouldn't that only affect the initial startup? Which indeed also is extremely slow but gets on its way once it's finished. How does that correlate to the continuous training speed? Is it reloading it each each time?

[bghira]

no, the VAE is not reloaded until validation runs and images generate. you'll have to profile the local setup to determine why it's slow

[XDM-Inc]

I'm no professional by far but I'll poke around and see what I can dig up. From a quick guess, I'm assuming there's just some setting enabled or something in the background that the other trainers do that's not either implemented or I have it set in simpletuner yet.