[New Feature] Integrated Openpose Editor

[huchenlei]

#1012 proposed an integrated openpose editor. Months passed, we are finally close to have one.

Workflow of this feature:
screen-capture.webm

There are still a few things that need to be worked out though:

How should the edited image be used by controlnet?

#1211 Discussed why we cannot make the preview result directly used by controlnet. TLDR is that many preprocessors use values from WebUI that are only available when clicking the generate button.

Option1: Let controlnet use pre-rendered openpose image if there is an update-to-date one. (Make an exception for openpose only as openpose preprocessor does not depend on any WebUI runtime values)
Option2: Add a button to push the preprocessor result to the input image and select none for preprocessor. This option is not so ideal as once sent, the pose becomes uneditable.

How should we release this feature to the users.

Behind the scene, the feature is embedding an openpose editor web page in an iframe. The controlnet JS code communiates with the iframe web page via window.postMessage. Currently this requires user install https://github.com/huchenlei/sd-webui-openpose-editor/tree/main separately.

If the extra extension is not installed, how should we handle the prompting:
Option1: Install the extension in the background. (Maybe too agressive?)
Option2: Display a line of text let user know the feature need another extension to function.

[lllyasviel]

wow great. how can js send data to gradio? what is the currently used method?

[huchenlei]

wow great. how can js send data to gradio? what is the currently used method?

There is a hidden gr.Textbox that accepts pose JSON from JS code.
gradio-app/gradio#2981 explains how to formulate an event so that gradio backend can recognize data update on gr.Textbox. (Not surprising that the issue was raised by AUTOMATIC1111 lol)

[lllyasviel]

wow sounds very a1111. will take a look soon

[2blackbar]

This is pretty great, is there a way to have a button to make entire hand and bones appear instead of enabling bones one by one ?

[huchenlei]

This is pretty great, is there a way to have a button to make entire hand and bones appear instead of enabling bones one by one ?

Yes, you can toggle visibility on entire hand, body, face. But invalid/missing keypoints are marked invisible by default, so if there are invalid keypoints, they are likely to be jammed at the same location.

I am working on the documentation on the editor repo. Hopefully that will clarify things.

[lllyasviel]

[huchenlei]

Need to install https://github.com/huchenlei/sd-webui-openpose-editor/tree/main first. (No need to build now as it automatically pull latest release binary from GitHub.)

[huchenlei]

Now we display a error screen to guide user if no editor is detected.

[huchenlei]

I think I now have answers to the 2 questions originally asked:

How should the edited image be used by controlnet?

Option2: Add a button to push the preprocessor result to the input image and select none for preprocessor.
Note: Option1 takes too much effort to implement and might cause confusion to users.

How should we release this feature to the users?

Option2: Display a line of text let user know the feature need another extension to function.

This PR is now ready to be merged.

[lllyasviel]

or perhaps we just add a dirty flag to preview image. if sha256 of a input image match some marked sha256 then replace it with another image before processing. maintain a sha to ndarray dict

[huchenlei]

or perhaps we just add a dirty flag to preview image. if sha256 of a input image match some marked sha256 then replace it with another image before processing. maintain a sha to ndarray dict

Probably need to hash all the input settings (resolution, etc) used to generate the preview image instead of just the hash of input image. I think for now a "Send back" button is the most cost efficient solution.

Note:
I find it really hard to navigate controlnet.py Script.process code. We probably want some sort of refactoring there soon. (Breakdown the logic of the big process function and add unittests to them.)

[lllyasviel]

then user click two buttons,first a button from extension and then send back?

[lllyasviel]

i think i can try it a bit tomorrow or later soon. how big is that extension? is it possible to directly move into cn, or hide the button if extension not installed?

[huchenlei]

i think i can try it a bit tomorrow or later soon. how big is that extension? is it possible to directly move into cn, or hide the button if extension not installed?

The extension is not too big. 2~3k lines of typescript code. The problem is that the extension needs to be build with nodeJS installed. My current workaround is to let user pull the compiled app from github release. If we include the editor in this repository, it is likely to messup the release page.

I can add an option in settings to hide the edit button, and prompt the user to go to settings and toggle that option if they don't want to see an edit button with alert sign.

[huchenlei]

then user click two buttons,first a button from extension and then send back?

When the user is satisfied with the image in the preview, the user can click Send back button to use the preview image as input image.

Overall workflow:

1. Click edit to pop up the editor modal
2. Do the necessary edits
3. Click Send to ControlNet in the editor modal to close the editor
4. Repeat step#1 if there are other changes to be made or click Send back to use the edited image as controlnet input image

[lllyasviel]

it seems when I click a button titled "Send to ControlNet", I will expect that this image will be used by CN. Clicking another button again sounds a bit ...
Perhaps I can try later and see if we can find a solution.

[huchenlei]

it seems when I click a button titled "Send to ControlNet", I will expect that this image will be used by CN. Clicking another button again sounds a bit ... Perhaps I can try later and see if we can find a solution.

We can directly set the input image when clicking send to ControlNet but that would overwrite the input image. If the user decides to adjust the pose again, the original input image won't be available as the background image in the editor.

The ideal way is let ControlNet use the edited pose in the generated image section, but I found it not so easy to implement.

[huchenlei]

@lllyasviel Gentle ping on the review. I can try to implement the logic to use the up-to-date preview image if you think that is the way to go.

[lllyasviel]

I am relatively busy this week. I think we can use a branch in this repo or what to test the features, and then perhaps the weekend I can test it more and try to merge hopefully.
Since the difference looks big and many users are actively updating this recently, I do not recommend to directly work on main branch for this.

[huchenlei]

I am relatively busy this week. I think we can use a branch in this repo or what to test the features, and then perhaps the weekend I can test it more and try to merge hopefully. Since the difference looks big and many users are actively updating this recently, I do not recommend to directly work on main branch for this.

A new branch embed_pose_edit is created.

[huchenlei]

Hi @lllyasviel,

Can you take a look again? I add a new checkbox to allow direct usage of preview image as controlnet input.

Now the workflow is reduced to:

1. Click edit to pop up the editor modal
2. Do the necessary edits
3. Click Send to ControlNet in the editor modal to close the editor
4. Click genereation to run SD or go back to Step 1 to do more edits

[huchenlei]

I am going to merge this if no further oppositions / feedbacks are received.

Some testings:

• Edit / Remove person
  edit_demo.webm
• Disable feature
  feature_off_demo.webm

We can either make the feature disabled by default or revert this PR if the user feedback is really negative.

[continue-revolution]

1. It would be better if you could add a feature where people could directly add and edit pose, without any reference image. (like posex, but posex cannot edit face/hand. Also editing face/hand should be optional)
2. It would be better if people do not have to build front end stuff. This is especially painful for non-experts like me.

[huchenlei]

1. It would be better if you could add a feature where people could directly add and edit pose, without any reference image.
2. It would be better if people do not have to build front end stuff. This is especially painful for non-experts like me.

1. The feature can be accessed directly in localhost:7860/openpose_edit_index. The default will not include hand/face.
2. People don't need to build front-end stuff as the extension downloads the compiled app zip from github release.

[huchenlei]

Also from the demo there is an issue with the openpose detection: The same hand got detected twice on the right person. Going to open an issue for this.