Implement framewise encoding/decoding in LTX Video VAE

[a-r-r-o-w]

Currently, we do not implement framewise encoding/decoding in the LTX Video VAE. This leads to an opportunity for reducing memory usage, which will be beneficial for both inference and training.

LoRA finetuning LTX Video on 49x512x768 videos can be done in under 6 GB if prompts and latents are pre-computed, but the pre-computation requires about 12 GB of memory because of the VAE encode/decode. This can be reduced by a considerable amount and lower the bar for entry into video model finetuning. Our friends with potatoes need you!

diffusers/src/diffusers/models/autoencoders/autoencoder_kl_ltx.py

Line 949 in d413881

raise NotImplementedError(

As always, contributions are welcome 🤗 Happy new year!

[rootonchair]

Hi @a-r-r-o-w, this is interesting and I would like to take it

[rootonchair]

Here it the result of my first attempt, If you notice there are still some inconsistency. I will try enhancing it if possible

No framewise decoding:

ltx_org_output.mp4

Framewise decoding:

ltx_output.mp4