GSOC 2026 Interest: Project #4 add image to video support to LTX Video Generation

[Yash-Vijay29]

Hello @likholat @sgonorov

Im Yash Vijay, a 3rd year university student enrolled in a dual program of Msc Mathematics and B.E Computer Science,
from Bits Pilani.

I mostly just wanted confirmation on what i understood from the project details:

So we already have a text to video pipeline that denoises noise into videos from text and are updating it so that it 1)inputs an image, 2) turns it into latent 3) denoises that based on text embedding and latent 4) gives out video preserving image structure.
Is that the problem statement or did i miss something?

Thanks for your time!

[Yash-Vijay29]

I went through the LTX pipeline and the transformer wrapper.

For image+text to video, I see two integration paths:

Add an image encoder and fuse image embeddings with text embeddings before passing them as encoder_hidden_states,
This would require the pretrained transformer to already support multimodal conditioning, i dont know if thats true.

Keep the transformer unchanged and initialize latents from an encoded image (via VAE) with added noise, so diffusion remains structurally identical but anchored to the image.

My understanding is that (2) is architecturally safer unless the underlying LTX model was trained for multimodal inputs.

Could you clarify whether the pretrained LTX transformer is expected to support multimodal conditioning, or if the intended approach is latent initialization?

[likholat]

Hi Yash,

Please note that the final proposal must be submitted through the GSoC portal webapp to be considered

Yes, your understanding of the problem statement is correct at a high level: the expected direction is to encode the input image into latents and use them together with text conditioning

As for the implementation details, we do not expect applicants to fully define the exact integration approach at this stage. The concrete technical details will be discussed further during the program.

Also, could you please attach the PR/PRs you have worked on in OpenVINO repository?

Thanks for the thoughtful questions and for taking the time to look into the pipeline.

Regards,
Anna

[Yash-Vijay29]

Yes i submitted a proposal with my current idea of what to implement.

Current PRs I worked on/working on:

• Guidance Rescale [merged]: Add guidance rescale [Divide by 0 check added] openvino.genai#3369
• Concat-reducemax Logic bug [merged]: Concat reducemax bug #33384
• Implementation for timestep_conditioning [to be reviewed]: Implement timestep_conditioning [ran over 10 samples] openvino.genai#3411
• Support LTX-Video 0.9.1 (timestep enabled models) [to be reviewed]: Support Export of LTX-Video 0.9.1 huggingface/optimum-intel#1652
• LTX-Video vae_factor_scale() formula correction [yet to be reviewed]: [Bug Fix] LTX-Video vae_scale_factor formula [Run checks] openvino.genai#3630
• Apply proper breakdown for aten:tensordot in Pytorch frontend [in review]: Support for aten::tensordot  #33337
• Bug fix for Prelu to handle 1d inputs [yet to be reviewed]: modified prelu for 1d #33388
• Bug fix for data corruption, implicit conversion of logic bits for different inputs [yet to be reviewed]: added implicit conversion logic for different bit inputs #33390
• Pytorch frontend operator additon: aten:fake_quantize_learnable_affine [yet to be reviewed]: Support for aten::fake_quantize_learnable_per_tensor_affine #33336

I am mostly working on getting timestep_conditioning to get working in the LTX Pipeline currently. I realized that optimum-intel itself doesnt export the Lightricks/LTX-Video-0.9.1 (the improved model over the base model with timestep_conditioning in its VAE_Decoder) so im writing a feature for that as well.

Adrian announced some pre requisite task for the proposal. If you have such in mind do let me know!