Add support for COCO style datasets for instance segmentation

[roboserg]

Feature request

Create a standard dataset loader capable of taking datasets in the JSON COCO style format and converting them into the Huggingface format. The DatasetDict will be generated with the correct features and configurations, making it suitable for various downstream tasks, such as instance segmentation fine-tuning with the Mask2Former mode from Huggingface hub.

The loader should include a flag that allows users to specify the type of segmentation they want to load, such as "panoptic," "semantic," or "instance." These different segmentation tasks store data differently within the COCO format. It is important to note that COCO segmentation masks can be represented in two ways inside the JSON file - either as a polygon or a bitmask.

Motivation

COCO style formatted datasets are prevalent in computer vision, especially in segmentation. Adopting them would ease the transition for Deep Learning practitioners as data preparation is often the main hurdle.

Your contribution

I am versed as to how the COCO format is structured but I am a total newbie to Huggingface.

[amyeroberts]

Hi @roboserg, thanks for opening this feature request!

transformers isn't responsible for datasets preparation, and so we wouldn't add a converter as part of the standard library.

The best place to put logic like this is in an example script or demo notebook.

• Scripts: https://github.com/huggingface/transformers/tree/b0f23036f16b961e14cf480522fef8581c4bf19c/examples/pytorch e.g.
• Notebooks: https://github.com/huggingface/notebooks/tree/main

This would provide a working example of how to transform a coco-style dataset

Ideally datasets should be converted and then uploaded to their DatasetDict equivalent on the hub. Perhaps a script would be useful to add so users could quickly convert and upload their own datasets?

cc @rafaelpadilla

[rafaelpadilla]

Hi @roboserg :)

I created a COCO dataset for bounding boxes only. Maybe it could be useful to you:
https://huggingface.co/datasets/rafaelpadilla/coco2017

You can find a COCODataset class, which takes the loaded_json dictionary representing the JSON containing COCO's bounding boxes. If you find it useful, you could adapt it for other cases (panoptic and semantic). This way, you can have a dataset class to represent COCO's samples and annotations used in a dataloader.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.