feat: Segment Anything Model Integration

[shondle]

This adds the ability to use Facebook's Segment Anything Model (SAM) with Label Studio.

Users can use a smart keypoint to generate a blush annotation with SAM for any object on an image, and adjust using tools already provided in Label Studio. This makes it much easier to create annotations quickly for any image segmentation use case.

For this, I created an ML back end that takes the image and smart KeyPoint a user places in Label Studio, and use it to generate the mask for the selected object with SAM. This is then converted to RLE and then passed to Label Studio to be generated as a blush annotation.

This has been tested using a computing cluster environment and with a CPU (after adjusting the device parameter) on a local machine.

[Aciid]

Hi, thanks for awesomeoness.
Quick suggestion, can it be modified so it works like in roboflow's segment-anything. Re-clicking anyhere adds to same segment, reclicking removes from segment.

Heres a few sample pictures, if you have not tried the one in roboflow, it probably uses "invisible to userinterface" -layer segments to handle this.

This makes segmenting really quick and not tedious, choosing when to add/substract from segment or to create new label could of course be using a modifier-key such as ctrl or a gui switch too?

Thanks for awesomeness

[iraadit]

Hi @Aciid and @shondle

Thank you for your work @shondle !

In https://www.trainyolo.com implementation, they use:

• left click to add to a segmentation
• right click to remove from this segmentation
• space to validate the segmentation (and start a new one)

[JaneX8]

I can't wait to try this. This would bring this tool to a whole next level. :)

[vansin]

Thanks for contribute to community~

We integrate it in OpenMMLab PlayGround , Welcome Star

Support Point2Label and BBox2Label

https://github.com/open-mmlab/playground/tree/main/label_anything

[shondle]

vansin, that looks awesome! All I ask is if you could credit my GitHub username for borrowing some of the code.

Also, I added a few more features that should lead to faster inference and a background eraser capability.

[vansin]

vansin, that looks awesome! All I ask is if you could credit my GitHub username for borrowing some of the code.

Also, I added a few more features that should lead to faster inference and a background eraser capability.

I add some information before,

no problem, I will add github username in openmmlab playground~

by the way , would you like contribute code to openmmlab playground ?

https://github.com/open-mmlab/playground/tree/main/label_anything

[erinmikailstaples]

Checking in on this — @KonstantinKorotaev, is there anything I can do to help get this across the line? 😊

[davidblom603]

+1 extremely interested in this

[ZhangYuef]

For "Eraser" feature: In my opinion, it should be used in adding and erasing smart keypoints for a same target label item.

[iraadit]

Hi
When will it be merged?
Thank you a lot for the work !

[makseq]

@shondle could you please check pytests and try to fix it in the new PR? Seems it was timeouted because a model tried to load too long.

[croche2574]

Could you add functionality to convert from bounding boxes to segmentations? This would be extremely useful for converting detection datasets to segmentation datasets for YOLO models

[shondle]

Could you add functionality to convert from bounding boxes to segmentations? This would be extremely useful for converting detection datasets to segmentation datasets for YOLO models

Here I added the ability to create the mask with a smart rectangle label. For converting an existing dataset you would need to change the box input x, y, width, and height by gathering from the tasks (what is already annotated on the image) instead of the kwargs. Then, you could just select all of the images and send them through the model.

Unsure about this, but may be better (for faster inference over a large set of images) to do this when I used the pytorch model this commit. instead of the ONNX (what I referenced earlier).

[shondle]

@shondle could you please check pytests and try to fix it in the new PR? Seems it was timeouted because a model tried to load too long.

I have tried to fix it, but running into issues with Docker (which I do not have much experience with). I fixed a few issues before this PR was merged, but was unable to figure out the fix for the timeout. If there is anyone with more knowledge working with Docker that is able to fix this issue and get over the hump, I would appreciate it. Otherwise, I will try to get back to this sometime later.

[Cupcee]

Hi, I tried installing this and everything seemed to go smoothly until I try to get a prediction:

...
def predict(self, tasks, **kwargs):
        """Returns the predicted mask for a smart keypoint that has been placed."""

        # Use this to check times for your predictions
        # print(f"Current data and time1: {str(datetime.datetime.now())}") # Current data and time1: 2023-04-16 18:56:09.361688 (ALMOST INSTANTANEOUS FROM THE RUN)

        results = []
        predictions = []
        predictor = PREDICTOR

        image_url = tasks[0]["data"]["image"]
        print(f"the kwargs are {kwargs}")
        print(f"the tasks are {tasks}")

        # getting the height and width of the image that you are annotating real-time
        height = kwargs["context"]["result"][0]["original_height"] # ERROR
        ...

It fails when getting the context from predict's kwargs. Context is None for me.

the kwargs are {'login': None, 'password': None, 'context': None}

Any idea on what goes wrong?

[KonstantinKorotaev]

Any idea on what goes wrong?

Did you add ML backend as interactive?

[Cupcee]

Did you add ML backend as interactive?

If you mean this switch, it's checked:

[KonstantinKorotaev]

If you mean this switch, it's checked:

Great!
Do you have Webhooks enabled?

[Cupcee]

Do you have Webhooks enabled?

How do I check if they are enabled and how do I enable them? Are they not enabled by default? The installation instructions mentioned nothing about this.

EDIT: I have this:

[KonstantinKorotaev]

Please deactivate option "Show predictions to annotators in the Label Stream and Quick View" in your Machine Learning settings for your project and check one more time.

[Marouene-Oueslati]

the the ML backend http address seems not to work properly..any fix for the SAM

[aiyou9]

Could you add functionality to convert from bounding boxes to segmentations? This would be extremely useful for converting detection datasets to segmentation datasets for YOLO models

I am also looking forward to this feature, especially in conjunction with this project

https://github.com/facebookresearch/ov-seg demo: https://huggingface.co/spaces/facebook/ov-seg

In this test example, vietanhdev/anylabeling#89

labeling other cells in the image can be manually corrected to the cells that need to be labeled and converted into instance segmentation, which greatly improves labeling efficiency.

another project:
https://github.com/luca-medeiros/lang-segment-anything

So, Image annotation can be combined with text prompt

[jackmead515]

Hi, I tried installing this and everything seemed to go smoothly until I try to get a prediction:

...
def predict(self, tasks, **kwargs):
        """Returns the predicted mask for a smart keypoint that has been placed."""

        # Use this to check times for your predictions
        # print(f"Current data and time1: {str(datetime.datetime.now())}") # Current data and time1: 2023-04-16 18:56:09.361688 (ALMOST INSTANTANEOUS FROM THE RUN)

        results = []
        predictions = []
        predictor = PREDICTOR

        image_url = tasks[0]["data"]["image"]
        print(f"the kwargs are {kwargs}")
        print(f"the tasks are {tasks}")

        # getting the height and width of the image that you are annotating real-time
        height = kwargs["context"]["result"][0]["original_height"] # ERROR
        ...

It fails when getting the context from predict's kwargs. Context is None for me.

the kwargs are {'login': None, 'password': None, 'context': None}

Any idea on what goes wrong?

Bumping this issue up. Seems like this is a persistent issue on my end as well. Is there a solution to this?

[DimIsaev]

Why is this problem not solved, it is 5-6 problems and still not solved!
Only messages that will be considered sometime...

[shondle]

Why is this problem not solved, it is 5-6 problems and still not solved!

Only messages that will be considered sometime...

Hi, are you using the smart keypoint tool on the toolbar while selecting one of the bottom two labels provided in the labeling config in the README?

Other things to check are whether Auto Annotation and Auto accept annotation are on in the image tab before you use the smart keypoint using one of the bottom labels to make a prediction.

Make sure "Use for interactive preannotations" is toggled on when you are adding your model. In the Machine Learning tab, make sure only the bottom toggle is activated aka "Show predictions to annotators in the Label Stream and Quick View".

[DimIsaev]

yes such a problem

[shondle]

yes such a problem

Once it has the smart keypoint selected can you click off to the side of the screen to unselect? If that doesn't keep it selected then if you need to toggle through the options just keep clicking the purple box in the toolbar not the sidebar.

[DimIsaev]

Once it has the smart keypoint selected can you click off to the side of the screen to unselect? If that doesn't keep it selected then if you need to toggle through the options just keep clicking the purple box in the toolbar not the sidebar.

1. it works illogically, or I don't understand the logic

2. But what about this error? She's more important
   feat: Segment Anything Model Integration #253 (comment)

Thanks