Can pretrained autoencoder stage be used for any new dataset training?

[Revist]

In the latent diffusion paper authors mention that:
"A notable advantage of this approach is that we need to train the universal autoencoding stage only once and can therefore reuse it for multiple DM trainings or to explore possibly completely different tasks."
Does that mean that we do not have to retrain autoencoder stage for image <-> latent space encoding / decoding when we want to train DM on a new dataset i.e. the autoencoder is general enough? This seems pretty strange to me.

I see that here:
huggingface/diffusers#356

vae = AutoencoderKL.from_pretrained( args.pretrained_model_name_or_path, subfolder="vae", use_auth_token=args.use_auth_token )

the vae is just loaded, which would seem that this is the case.

[gombru]

Same doubt here.

Although I do assume that the autoencoder is general enough, might be better to retrain it for some domains.

In the paper they don't specify any autoencoder training details. In the code it seems however they train it in OpenImages and not in LAION. Is this correct?

• If that's the case, wouldnt the autoencoder benefit from more training data?
• Also, how expensive is it to train it? More details about that would be helpful!

[forever208]

@gombru 2 years ago, I got an answer from the author of a concurrent work (Vector Quantized Diffusion Model for Text-to-Image Synthesis) to LDM. He said training AE on ImageNet or OpenImages is general enough, but training the AE on the target dataset yields the best performance.