[1.1.236] Support DW Openpose preprocessor

[huchenlei]

See initial issue here: #1855

DW Openpose preprocessor greatly improves the accuracy of openpose detection especially on hands. Here is a comparison used in our unittest:

Input Image:

Openpose Full Output (Right hand missing):

DW Openpose Full Output:

Usage:

With this pose detection accuracy improvements, we are hyped to start re-train the ControlNet openpose model with more accurate annotations. There is a proposal in DW Pose repository: IDEA-Research/DWPose#2.

I would really want @lllyasviel to take the initiative for this retraining task, but he probably busy with other tasks. I am not familiar with ControlNet training, so I welcome suggestions and ideas on how should we proceed on this task with community support. You can find me on my discord server: https://discord.gg/GkaWcUat7R. Or just leave a comment here.

[huchenlei]

Note:

According to #1864 (comment), you need to install Microsoft Visual C++ (https://visualstudio.microsoft.com/visual-cpp-build-tools/) for some dependencies to build correctly.

[Erwin11]

run DW Openpose preprocessor ,it shows [TypeError: 'NoneType' object is not iterable]
version: ControlNet v1.1.237, SD v1.5.1,

full code:

Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: v1.5.1
Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a

current transparent-background 1.2.4

Installing sd-dynamic-prompts requirements.txt

Installing requirements for TemporalKit extension

Launching Web UI with arguments: --medvram --opt-sdp-no-mem-attention --api
No module 'xformers'. Proceeding without it.
[-] ADetailer initialized. version: 23.7.9, num models: 9
2023-08-05 08:10:39,593 - ControlNet - INFO - ControlNet v1.1.237
ControlNet preprocessor location: E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-08-05 08:10:39,758 - ControlNet - INFO - ControlNet v1.1.237
sd-webui-prompt-all-in-one background API service started successfully.
Image Browser: ImageReward is not installed, cannot be used.
Loading weights [5493a0ec49] from E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\models\Stable-diffusion\ACG\AOM3A1B_orangemixs.safetensors
Creating model from config: E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\configs\v1-inference.yamlLatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors
Applying attention optimization: sdp-no-mem... done.
Model loaded in 3.2s (load weights from disk: 0.3s, create model: 0.5s, apply weights to model: 0.6s, apply half(): 0.7s, load VAE: 0.2s, load textual inversion embeddings: 0.2s, calculate empty prompt: 0.7s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
[AgentScheduler] Task queue is empty
[AgentScheduler] Registering APIs
Startup time: 87.5s (launcher: 38.9s, import torch: 4.0s, import gradio: 1.7s, setup paths: 1.8s, other imports: 1.7s, load scripts: 15.6s, create ui: 5.6s, gradio launch: 17.8s, app_started_callback: 0.3s).
2023-08-05 08:12:37,515 - ControlNet - INFO - Preview Resolution = 512
Loads checkpoint by http backend from path: https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_l_8x8_300e_coco/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth
Loads checkpoint by local backend from path: E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\openpose\dw-ll_ucoco_384.pth
Traceback (most recent call last):
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1323, in process_api
    result = await self.call_function(
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1051, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet_ui\controlnet_ui_group.py", line 606, in run_annotator
    result, is_image = preprocessor(
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 79, in decorated_func
    return cached_func(*args, **kwargs)
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 67, in cached_func
    return func(*args, **kwargs)
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\global_state.py", line 35, in unified_preprocessor
    return preprocessor_modules[preprocessor_name](*args, **kwargs)
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\processor.py", line 265, in run_model
    return remove_pad(self.model_openpose(
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\openpose\__init__.py", line 360, in __call__
    poses = self.detect_poses_dw(oriImg)
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\openpose\__init__.py", line 337, in detect_poses_dw
    return Wholebody.format_result(keypoints_info)
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\openpose\wholebody.py", line 103, in format_result
    body = BodyResult(body_keypoints, total_score(body_keypoints), len(body_keypoints))
  File "E:\VM\test_model\stable-diffusion-webui-1.4\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\openpose\wholebody.py", line 93, in total_score
    return sum(keypoint.score for keypoint in keypoints if keypoint is not None)
TypeError: 'NoneType' object is not iterable

[huchenlei]

Fixed.

[Erwin11]

IT WORKS! Thanks for your hard work! small hands is more accurate

DW Openpose

Openpose

[huchenlei]

Due to #1875, the feature cannot be merged to main yet. I uploaded the version before reset to dw_pose_reland branch. If you still want to use the feature with a local compile of mmcv stuff, you can still switch to the dw_pose_reland branch for now while we work for a solution.

[p0mad]

@huchenlei
Thanks fo this discussion, i have Controlnet 234 installed on my win pc.
is that enough to just checkout to the dw_pose_reland branch and its good to go? or it also needs some other tweaks?

SO as you said before the dw pth model was not intended to be used for auto1111. so i guess just git checkout would needed for this purpose, right? ( or any other better ways recommanded?)

Thanks

[C0nsumption]

Hey everyone. Where do I download the preprocessor?

EDIT:
forget it. Only fetched. forgot to checkout to the branch.

[huchenlei]

The feature has relanded on main. You don't need to checkout other branches now.

[C0nsumption]

I did for Vlad but all good now. Works fine on Auto1111 main

[vararth]

If I understand correctly, dw_openpose_full is the same as openpose_full? If not, how can I set up the dw_openpose_full?

[viccho]

Hi everyone.
i'm new here and new on SD. I nedd to use dw pose but i can't reach the goal. here is a printscreen , the preprocessor detect face, but the result did not follow the preproc. have you an idea of what i can do ? (my prompt is just " girl" )

thnak you :)

[zainmalik95]

I'm using v1.1.410

[zainmalik95]

I've tried the dw openpose as well but still not able to get the desired results.

[huchenlei]

For your case, you are using SD1.5 ControlNet model to control SDXL base model.

Note: currently for SDXL controlnet models, none of them supports openpose full.

[zainmalik95]

Oh okay, which ones should i use? I actually have to cases to handle, one is change the face posing and the other is full body posing, what combination do you think will be the best for me? I'm actually learning so don't have the concepts clear as of now.
Thank you for the quick reply btw.

[huchenlei]

See #2039 for SDXL support details. Side note: updating the extension to v1.1.411 will enable Edit button feature that allows you to edit the pose in place.