Feature request: Object selection using Segment Anything

[patrikzudel]

It would be very cool if Photopea would have Object selection that is more comparable in performance to the one in Photoshop. Select subject is a good feature but lacks control, the user doesn't always want to select only the subject.

An approach that came to my mind is using the Segment Anything model locally in browser, there are two versions, mini (50mb) and base (300mb). Embedding of 1080p image on mini (AMD 5600X) takes 11 seconds. After an embedding is done, selection takes 0.1 seconds. On the base model embedding takes 6x longer.

Examples of local in-browser implementation:

• https://huggingface.co/spaces/radames/candle-segment-anything-wasm
• https://github.com/sunu/SAM-in-Browser

The implementations above have selection that's worse quality than the official MetaAI showcase available here
The Meta showcase has amazing selection and is a great showcase of the models power.

I am a paying user and I use Photopea professionally and love it. I would gladly even pay credits for good object selection if local execution of the model wouldn't be suitable. Thank you for considering this.

Segment Anything uses the Apache 2.0 License.

[patrikzudel]

Also found this WebGPU Implementation example.

https://github.com/microsoft/onnxruntime-inference-examples/tree/main/js/segment-anything

With WebGPU the speed is amazing, 1 second encoding for the base model (RTX3080).

[photopea]

I tihnk Segment Anything is very bad for object selection, as it can not detect precise borders of objects.

Here is what I got with Segment Anything (375 MB downloaded, 60 seconds processing on my CPU)

And here is what I got with Select - Magic Cut (part of Photopea, which is 1 MB) in about 5 seconds:

If you want a one-click solution, use Select - Remove BG. It uses a 10 GB AI model (on our server) and gives better results than Adobe Photoshop or any other software.

[patrikzudel]

I don't think this comparison is fair for usecases that are more complex, where Segment Anything would shine. Of course for main subject background removal the Remove BG would be far superior, but that is not the only usecase where object selection is needed. Let me give some examples where Segmentation is helpful.

Example image

For example I might need to change the subjects hair. 1 Click with segmentation.

I might need to extract only his right hand. 1 Click again.

I feel jagged edges could also be smoothed if needed. Here is an example of the glove smoothed using Photopea selection smoothing. This one is great.

As I've stated in my first post, I agree that the huggingface implementation of selection is bad, but the Meta demo implementation is good.

Here is a better comparison in my opinion.

Source Image:

Magic Cut (1 minute of work)

Remove BG (Good quality, but no control over what the "subject" is)

Segment Anything (Meta demo)

Segment Anything (Smoothed + Feather)

With WebGPU, Segment Anything embedding can take 1-3 seconds.

[photopea]

I think the Segment Anything + smoothing is a bad result for most of the people. The seleciton is smooth, but not precise. Also, I am afraid most of the people would not want to wait minutes to download 375 MB, they would rather use the lasso tool for two minutes.

[patrikzudel]

Yes, but let's not forget that they'd call the Magic Cut result worse.
Also, calling it bad needs context to say what it is bad for - for Generative Fill it's a good result, for making stylized cutouts it's a good result. If more precision is needed, the selection can still be post processed significantly, still saving time.

Also, I doubt that anyone would say that the Glove example with Segment Anything + Smoothing is bad.

It is much more important to focus on the real-world examples of use cases I presented (hair, glove, face etc.) rather than the bear which was a stress test. Not every selection object is going to have hair.

Regarding the download, the ONNX implementation I linked showcases a quantized model that is only 108MB in size, that isn't going to be minutes to download for most users, and internet is only going to get faster. Also, it is only a one time download as it can be cached in storage.

[patrikzudel]

My main point is that pursuing Object selection would be great, maybe another approach could be using Segment Anything (smoothed) inside Magic Cut - clicking on an object would set it as Foreground and everything else as Background, then the user could fine tune it using the brushes in Magic Cut if needed.

Here I used an approach where I combined Segment Anything + Refine Edge (Right side) + Remove BG (for left side) and made this bear in 1 minute, much more comfortably than masking it using the lasso tool.

[camoody1]

I agree that Segment Anything would be a wonderful addition to the Photopea tool set. But I also understand what Ivan is concerned about with the 320MB download. Perhaps making the tool available with a pop-up window making the user aware of a one-time download of 320MB to make it work? IS it a one-time download? Or would that download have to happen every time it was used? If you could make something like that a one-time download, I think it would be great to have.