-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request] Integrated openpose editor #1012
Comments
I couldn't agree with you more, |
There is also a openpose editor for hand specifically: Currently we live in a very chaos space where there are multiple extensions available, but none can do the task very well. |
openpose editor requires a javascript expert, where I am unfortunately not. I believe we need a JS expert to do this for us. |
I can help with the JS code. The JS implementations in the above extensions except the 3D extension are pretty minimal (<1k lines of JS). |
I kinda sort out the data flow to achieve the functionality: Build a separate sd-webui extensionThe extension will use the sd-webui api to expose a def mount_openpose_api(_: gr.Blocks, app: FastAPI):
@app.post('/openpose_editor', response_class=HTMLResponse)
async def index():
return templates.TemplateResponse('index.html', {"request": request"})
script_callbacks.on_app_started(mount_openpose_api) Embed the openpose extension's page into controlnet
window.parent.postMessage("Data from the child iframe!", "*");
window.addEventListener("message", function (event) {
console.log("Message received from the child iframe:", event.data);
// Simulate a click event on a button, so that ControlNet's python backend can receive the event. data is passed through
// gr.State.
state.textContent = event.data;
button.click();
}); ConclusionBy doing this we do not need to host the code of openpose editor within controlnet. User can also choose any openpose editor that support the controlnet's message protocol. AlternativeWeb UI uses very clumsy pure |
Building a pure front-end openpose editor here: |
Often times the openpose preprocessor cannot produce the exact openpose we want. Here are some experience I have:
I propose integrate an openpose editor into controlnet extension.
Here are some existing works:
The first 2 extensions copy controlnet's
annotator/openpose
directory to detect pose from image. Non of them has updated the code to support hand/face detection yet. Both extension supports sending the image to be used by controlnet.The 3D editor feels too complicated to use, as moving an anchor point in 3D is way more complicated than in 2D.
Proposed workflow:
Edit Pose
button is shown below the generated image when any openpose model is selected as preprocessor.Edit Pose
button will send the JSON openpose data (Stored somewhere on server side when running the preprocessor?) to the openpose editor (A modal?)I am not so sure what is the best way to hijack the openpose JSON data on server side. Please provide some ideas, thanks!
The text was updated successfully, but these errors were encountered: