[LTX0.9.5] make LTX0.9.5 works with text-to-video

[yiyixuxu]

see more context here #11143 (comment)

[yiyixuxu]

I will work on this but if someone in the community can pick this up sooner we'll really appreciate it too!

[nitinmukesh]

If someone take this up

All checkpoints (0.9.0, 0.9.1, and 0.9.5) use the same transformer architecture, but their VAEs differ.

Lightricks/LTX-Video#128 (comment)

and requesting to support both LTXPipeline and LTXImageToVideoPipeline for 0.9.5.

1. This is the only model that I know which works on low VRAM without any quantization, GGUF, etc... and produces decent quality.
2. This is the only model which when spilled on shared RAM does not increase processing time significantly (1 min may become 5 min) while with all other models I tried (1 min becomes 30 mins or more)

[bghira]

is there more info on why it is broken?

[yiyixuxu]

and requesting to support both LTXPipeline and LTXImageToVideoPipeline for 0.9.5.

any reason you need that if you're able to run everything with LTXConditionPipeline?

[yiyixuxu]

@bghira it's not broken, we made a separate pipeline (LTXConditionPipeline) for 0.9.5 to in order to have a very flexible support for all the conditions it supports

[bghira]

@yiyixuxu i guess i poorly phrased my question, the title of this issue is to "make LTX0.9.5 works with text-to-video" but i'm curious what work is required to make it work, if it's not broken currently this is confusing to me.

additionally, the LTXPipeline does not stop the user from using it for LTX0.9.5, is it supposed to?

[nitinmukesh]

any reason you need that if you're able to run everything with LTXConditionPipeline?

I don't need it, if everything works with LTXConditionPipeline. I thought for the sake of consistency (considering previous version) this is how it will be done, wasn't sure that LTXConditionPipeline can have everything.

[yiyixuxu]

I thought for the sake of consistency (considering previous version) this is how it will be done, wasn't sure that LTXConditionPipeline can have everything.

it would be easier for us to support from LTXCOnditionPipeline since they made quiet a lot of changes for 0.9.5. Agree it is not very consistent but the use case it supports (arbitrary combination of text/image/video) does not fit into any of the current pipeline anyway so we decided to make a new one

[yiyixuxu]

@bghira

i'm curious what work is required to make it work

I think it should be easy to add support for text-to-image, e.g. prepare_latents return a noise, latents is packed without these additional condition logic etc, but won't be sure until we test it out. (we are having diffusers offsite this week so we are a bit slow.)