[Feature]: Stable Video Diffusion from SAI

[uk9921]

Expected behavior

stable-diffusion released img2video model. Do you have plan to support it?

[continue-revolution]

I've already noticed this model. I do have plan to support it in some way. Please wait patiently.

[drhead]

I have actually tested using the special VAE decoder from the SVD model with AnimateDiff outputs and it helps output quality quite a bit, removing a lot of noise. That could be implemented relatively easily and bring a great benefit.

[continue-revolution]

Interesting. @drhead Do you mean replacing the original VAE with SVD VAE?

[uk9921]

@drhead Can you elaborate more on this? Do you input the latents generated by Animatediff into the VAE decoder of SVD?

[drhead]

Interesting. @drhead Do you mean replacing the original VAE with SVD VAE?

Latents output from AnimateDiff can be decoded using the decoder half of the SVD VAE. They are then decoded with temporal awareness, which eliminates a large amount of noise in the output. I have only been able to do a modest amount of testing and have only tested it on realistic-style videos (since lack of training data may mean there will be issues decoding animation-style videos), but from what I can tell so far it does work rather well. Main issue is porting the model out of the SGM codebase.

[continue-revolution]

@drhead it would be helpful if you could share some code with me

[uk9921]

@drhead Very reasonable. We are very much looking forward to you sharing some examples of the differences before and after using SVD-VAE.

[zhangrc]

Is my 3060 still sufficient? Is it necessary to upgrade the GPU

[andzejsp]

when? world is waiting

[uk9921]

@zhangrc We can run with 6GB VRAM now, ref to this: https://github.com/brycedrennan/imaginAIry

[uk9921]

f5 is broken

[continue-revolution]

this is now available via Forge. Unfortunately it is too hard if we want it on OG A1111.