Use VAE for video compress causes OOM

[CacacaLalala]

I fisrt use VQVAE for video compress, the code runs fine. But Loss drops very slowly.

So I change AE to VAE, and I got the OOM error, even though I set batch_size and accumulation_steps to 1.

Has anyone encountered this problem too?

[zhengzangw]

Now we use VAE for training. To lower the memory, you can set micro_batch_size in the inference config.