Custom Segmentation

[Dicko87]

Hi there @alcinos I have had great results using DETR in object detection but would like to extend this to segment the detected objects. How do I modify my existing object detection model for instance segmentation?

Thank you

[abdksyed]

@Dicko87
https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/DETR_panoptic.ipynb

[Dicko87]

@abdksyed thank you very much for the link, I have seen this already and was really wondering how to set up the json files, their structure , where to put the one files etc and what command to use to get training going. I am now in a position to start playing around and try to get this working. I’ll post back with an update. Thanks again 😊

[Dicko87]

I have a json file where the segmentation_info are bounding box coordinates. I also have a folder of mask images (png files) . I am wondering if I need to use coco_panoptic.py for this data, because the normal coco.py file takes polygon masks from the segmentation info in the json and so it would not be able to give the propper masks with just bounding box coordinates, it will need the information from the png images.

[Dicko87]

Hi guys, I've ran into a problem with using segmentation, any help would be appreciated. thank you.
line 78, in rgb2id return int(color[0] + 256 * color[1] + 256 * 256 * color[2]) TypeError: only size-1 arrays can be converted to Python scalars

I have my panoptic_coco (custom) dataset where each image contains multiple masks.
The json file has one annotation per image, describing the bounding box for each mask as shown in the detr examples.
I'm not sure what is wrong.

I think it is something to do with: https://cocodataset.org/#format-data, Panoptic Segmentation, item 2 which says:
For each annotation, per-pixel segment ids are stored as a single PNG at annotation.file_name. The PNGs are in a folder with the same name as the JSON, i.e., annotations/name/ for annotations/name.json. Each segment (whether it's a stuff or thing segment) is assigned a unique id. Unlabeled pixels (void) are assigned a value of 0. Note that when you load the PNG as an RGB image, you will need to compute the ids via ids=R+G256+B256^2.

[abdksyed]

The annotation folder must also have the image of the mask. As per the cocodataser.org/#format-data #4 which is Panoptic Segmentation, if you see they mention that the PNG images must be saved in the annotation folder, with the same name corresponding to the input image.

If the image is say img_001.jpg, in the image folder, in the annotation folder there must be a same image_001.png which should be the mask of the image.

[Dicko87]

Hi folks, I am looking for some help please.

I am running DETR for segmentation and have ran into the following problem when running the model.

[abdksyed]

You would have to make coco evaluator as None line in engine.py

https://github.com/facebookresearch/detr/blob/main/engine.py#L77

[abdksyed]

I did train DETR on Custom Dataset,
https://github.com/abdksyed/DETR
This will have almost all the steps, you can follow it, look into repo what all changes I had to do and all.

You can open any issue there, if you need help.

[Dicko87]

Hi @abdksyed, thank you very much for your reply. So I would just need to comment out line 77, the same as line 78 has been commented out via #.

Thank you very much for the link, I will take a look and will be in touch :)

[abdksyed]

https://github.com/abdksyed/DETR/blob/main/detr/engine.py

Yeah you can look into these, what to comment out.

[Dicko87]

Brilliant @abdksyed, I will take a look and make the changes.
Thank you very much for your help. I'll be back with an update. :)

[Dicko87]

Hi there @abdksyed , I commented out the lines but I am still facing the same error (identical) as shown above.