AnimateDiff in Diffusers Library

[aandyw]

Model/Pipeline/Scheduler description

I think it would be cool to work on turning AnimateDiff into a diffusers pipeline that can be used via the library.

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Paper: https://arxiv.org/abs/2307.04725
Github: https://github.com/guoyww/AnimateDiff

[aandyw]

I would like to work on this pipeline if it's something that people are interested in.

[DN6]

@pie31415 go for it 👍🏽 Feel free to add me to do a review and let me know if I can help.

[aandyw]

@DN6 Sounds good. I'll have time to work on this in a few weeks. I'll ping you once something is up.

[DN6]

@pie31415 Looking at the AnimateDiff repo, it looks like it's adding an additional motion module to the 3D Conditional Unet. This is a similar paradigm to Adapters and Controlnets. It might be a route worth investigating when trying to create the Motion Module.

[haofanwang]

Agree to @DN6 , this absolutely worths a shot.

[DN6]

@haofanwang Would you be interested in collaborating on this? Would be great to get your eyes on a review.

[aandyw]

Yeah, I would definitely appreciate all the help/suggestions. These next 2 weeks will be a little busy because of exams but after that I'm really excited to get started and working with you all.

[charlesmccarthy]

awesome to see it count me as interested

[Skquark]

I integrated AnimateDiff into my project https://DiffusionDeluxe.com using the modified CLI version. The official one is pretty outdated, made for an old Diffusers version and Torch < 2.0 which made it unusable, so I found a refactored fork by Neggles at https://github.com/neggles/animatediff-cli which is much better coding. He even got rid of the 16 frame limit, which was essential for animating anything of value. I'd suggest using that code as the base of the AnimateDiff Pipeline within Diffusers, would save some time reworking it...

[notdanilo]

Any update on this? I would love to try it :)

[DN6]

Hi @pie31415. How is the PR progressing for this?

[DN6]

@notdanilo Please feel free to go ahead here