How to train Stable Video Diffusion model?

[howardgriffin]

No description provided.

[ersanliqiao]

sorry，ask me

[Angelalilyer]

sorry，ask me
Hello！ Do you have a training script for Stable Video Diffusion? Could you please contact me

[shijianjian]

find a non official one #267

[hadipash]

Our team has released the SVD training script:
https://github.com/mindspore-lab/mindone/tree/master/examples/svd
It's still under development, but you can use it for reference.

[bdytx5]

What are the GPU requirements for fine tuning @hadipash ?

[hadipash]

What are the GPU requirements for fine tuning @hadipash ?

Currently a lot (64 GB), but we're working on reducing VRAM usage.

[bdytx5]

@hadipash thats not bad really. You tested it with sharding across multiple GPU's (eg multiple GPU's with <64gb of VRAM)?

[hadipash]

@hadipash thats not bad really. You tested it with sharding across multiple GPU's (eg multiple GPU's with <64gb of VRAM)?

Currently, a sequence of 4 frames can be trained on a single 64GB GPU. We are working on 1) optimizing memory usage on a single device and 2) implementing distributed training to allow for longer sequences (e.g., 30+ frames).

[DataAIPlayer]

@hadipash hello，I tried using LoRA to fine-tune the U-Net with SVD, and even with a batch size of 1, memory overflow occurs on the A100 GPU when the dataset consists of 25-frame videos. Does this mean that model parallel training must be employed, distributing the model parameters across multiple GPUs?

[hadipash]

@DataAIPlayer Not sure about LoRA, haven't integrated it yet. However, for vanilla training - yes, need distributed training as a single 64GB GPU can only fit 4 frames.