ClipSeg workflow for capturing backgrounds elements ideas

[ac-pill]

Hi guys,

I'm trying a series of custom nodes for Clip Segmentation based on prompt outputs.

My use case is: trying to create a series of masks for elements (chairs, lamps, tables, couches) in the background images I've created, with the maximum precision I can get. I'm using only a series input images for posterior Inpainting.

Currently the closest to what I have is WAS Node Suit Clip Segmentation Masking node + Mask Dominant Region Threshold, that get roughly the element. Precision is lacking thou.

Does anyone has similar experiences on that field?

Some potential ideas, I will try out:

• Combining the output results on a mask, convert to a series off matrices points and injecting back on SAM Segmentation masks as it has more edge detection precision
• Or training a model on a database of masks my own

Would love to get some ideas from the community here.

Thanks.

[ac-pill]

If anyone is interested, I found a way to workout the mask for better results:

• combining clip segmentation + sam segmentation masks: 1. getting the clip seg mask, transforming to a binary mask, and converting to segments using Impact Pack nodes to select the SAM mask region.

Results were pretty detailed, if anyone is interested I can paste the workflow here.

[deitylink1]

definitely interested, but also not sure about how you are working sam segmentation into a workflow? is there a custom node I don't know about? #nooblife

[ac-pill]

About SAM model yes, you will need:
https://github.com/WASasquatch/was-node-suite-comfyui
https://github.com/ltdrdata/ComfyUI-Impact-Pack

I used part on WAS nodes for mask dominant region control.

After installing you will be able to see the SAM loader and related nodes.
Here is a screenshot on the workflow, this is working really well.

Now my goal is to work on reverse, and get the segmentation back into element names, if you are testing something similar let me know.

[deitylink1]

Thanks for your reply - I am working on stuff on the side so I don't have great time to dedicate at the moment, but the biggest problem is installing was-node-suite-comfyui at the moment, then I can try what you've suggested.

Not that you have to be my debugger, but I seem to have a problem where somewhere in the install it can't figure out where my MS compiler is (windows 10 environment)

//////
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [47 lines of output]
The Meson build system
Version: 0.62.2
Source dir: windows temp (I've sanitized this line :) )
Build dir: windows temp (I've sanitized this line :) )
Build type: native build
Project name: SciPy
Project version: 1.9.1
Activating VS 17.6.4

  ..\..\meson.build:1:0: ERROR: Compiler cl can not compile programs.

A full log can be found at [Dir that doesn't exist in TEMP, so I CAN'T check the full log XD]

[deitylink1]

I think I got the was-node-suite installed, but I had to run through the "Requirements" manually to do it... odd... (ie open up the requirements and manually run a "pip install -r" for everything.

[ac-pill]

Hi @deitylink1 yes, some nodes require manual installation as some of them require OS libraries that are not installed on ComfyUI on the first place.
From my experience, any AI tools, models, scripts, use them inside a Linux docker container so you can automate or even preserve your OS overall.

If you made it work let me know.