[Test] new controlnet (animatediff_controlnet))

[s9roll7]

sample_animatediff_controlnet2.mp4

original / animatediff_controlnet + controlnet_openpose(0.5) / animatediff_controlnet + controlnet_openpose(0.5)
1024x1280

animatediff stylize create-config PATH_TO_MOVIE_FILE -gh
animatediff stylize create-region PATH_TO_STYLIZE_DIR
animatediff stylize generate PATH_TO_STYLIZE_DIR

[s9roll7]

sample_animatediff_controlnet3.mp4

original /
animatediff_controlnet + controlnet_openpose(0.5) + controlnet_normalbae(0.5) + ip_adapter/
animatediff_controlnet + controlnet_openpose(0.5) + controlnet_normalbae(0.5) + ip_adapter
768x1152

animatediff stylize create-config PATH_TO_MOVIE_FILE -gh
animatediff stylize generate PATH_TO_STYLIZE_DIR

[KaustubhPatange]

Hey @s9roll7, tried your exact commands but it seems like the background is not generated properly and looks green a bit. Here is my
prompt.json. What could be wrong?

Also it seems like animatediff_controlnet did not ran and it requires some conditioning images. What preprocess images have you supplied for openpose?

Original	Generated
input_.mp4	output_.mp4

[s9roll7]

Hey @s9roll7, tried your exact commands but it seems like the background is not generated properly and looks green a bit. Here is my prompt.json. What could be wrong?

Also it seems like animatediff_controlnet did not ran and it requires some conditioning images. What preprocess images have you supplied for openpose?

Original Generated
input_.mp4
output_.mp4

The json file looks fine.
I'm guessing 00_controlnet_image\animatediff_controlnet is empty?
If there's nothing in the directory, nothing will happen.
You need to copy all the images in controlnet_openpose to animatediff_controlnet.

[s9roll7]

sample_animatediff_controlnet4.mp4

original / animatediff_controlnet + controlnet_openpose
896x768
generation time 196sec (4090)
vram 10.3 GB

animatediff stylize create-config PATH_TO_MOVIE_FILE -gh
animatediff stylize generate PATH_TO_STYLIZE_DIR

[dancemanUK]

use the animatediff_controlnet , 4090,speed too slow, share DDR4 RAM 6gb.not in 4090 24GB VRAM

[s9roll7]

use the animatediff_controlnet , 4090,speed too slow, share DDR4 RAM 6gb.not in 4090 24GB VRAM

    "controlnet_map": {
        "input_image_dir": "..\\stylize\\2023-12-06T13-02-02-sample-mistoonanime_v20\\00_controlnet_image",
        "max_samples_on_vram": 200,           # <-----------
        "max_models_on_vram": 3,                # <---------
        "save_detectmap": true,

[dancemanUK]

use the animatediff_controlnet , 4090,speed too slow, share DDR4 RAM 6gb.not in 4090 24GB VRAM

    "controlnet_map": {
        "input_image_dir": "..\\stylize\\2023-12-06T13-02-02-sample-mistoonanime_v20\\00_controlnet_image",
        "max_samples_on_vram": 200,           # <-----------
        "max_models_on_vram": 3,                # <---------
        "save_detectmap": true,

thank you！

[AivoGenX]

Hello, I used the img2img function in Stable Diffusion 1.5, which only added prompt words. The effect is quite different from that of Animatediff cli prompt travel. How can I maintain better consistency with SD? Thank you

[aycaecemgul]

hi, can you share your json for this style? @s9roll7