Image Segmentation pipeline #13828

mishig25 · 2021-10-01T09:23:48Z

What does this PR do?

TLDR

Implements image-segmentation pipeline (for DetrForSegmentation atm).

API specifications

Input: image source (identical to the input of image-classification & object-detection pipelines)
Output:

List(Dict(
    mask: str, # base64 str
    label: float,
    score: int,
))

Two design choices I've made and would like to discuss (& modify if needed):

Output png_string has masks information using same mechanism as COCO panoptic segmentation annotations. See section 4. Panoptic Segmentation from https://cocodataset.org/#format-data. Paraphrasing a bit:

per-pixel segment ids are stored in the PNG string. Each segment is assigned a unique id. Unlabeled pixels (void) are assigned a value of 0. Note that when you load the PNG as an RGB image, you will need to compute the ids via ids=R+G256+B256^2.

Image segmentation pipeline accepts subtask arg. There are different variations of segmentation task (semantic, instance, panoptic, etc. see image below). If a model doesn't implement requested subtask, it gets defaulted to what's available. See example below:

transformers/src/transformers/models/detr/feature_extraction_detr.py

Lines 738 to 739 in dd5c269

    
           logger.warning("No subtask was supplied, defaulted to panoptic") 
        
           return self.post_process_panoptic(outputs, processed_sizes, threshold=threshold)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Discussed in Tracking integration for Image Segmentation hub-docs#43, New object detection and image segmentation widgets hub-docs#6, Widget Image Segmentation huggingface_hub#378
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines,
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Update 1:

Updated output shape:

List(Dict(
    mask: str, # base64 str
    label: float,
    score: int,
))

The goal for any segmentation architecture is to implement the most detailed version of segmentation subtask they can so that other subtasks can be reconstructed (if needed). For example, if an architecture post_process_segmentation method implements part-aware panoptic, other subtasks (including semantic, instance, etc.) can be reconstructed from part-aware panoptic output since all the details needed are in there

Narsil

Very nice PR!

I think we can add more representative power to the pipeline output removing the need for subtask or at least internalize it's complexity (so we don't force model implementors to add irrelevant stuff)

The tests can be simplified quite a bit IMO.
If we really want subtask to exist, we need test that display the differences they imply.

Happy to help on changing the representation if needed.

Narsil · 2021-10-04T09:42:02Z

src/transformers/pipelines/image_segmentation.py

+    @staticmethod
+    def load_image(image: Union[str, "Image.Image"]):
+        if isinstance(image, str):
+            if image.startswith("http://") or image.startswith("https://"):
+                # We need to actually check for a real protocol, otherwise it's impossible to use a local file
+                # like http_huggingface_co.png
+                image = Image.open(requests.get(image, stream=True).raw)
+            elif os.path.isfile(image):
+                image = Image.open(image)
+            else:
+                raise ValueError(
+                    f"Incorrect path or url, URLs must start with `http://` or `https://`, and {image} is not a valid path"
+                )
+        elif isinstance(image, Image.Image):
+            pass
+        else:
+            raise ValueError(
+                "Incorrect format used for image. Should be a URL linking to an image, a local path, or a PIL image."
+            )
+        image = image.convert("RGB")
+        return image


It might be time to create a utils file and put it in there maybe so we can just reuse this code all the time and test it separately. (it should be another PR, just mentioning it here)

@LysandreJik Would you agree ?

src/transformers/pipelines/image_segmentation.py

tests/test_pipelines_image_segmentation.py

src/transformers/pipelines/image_segmentation.py

tests/test_pipelines_image_segmentation.py

NielsRogge · 2021-10-04T17:59:16Z

src/transformers/models/auto/modeling_auto.py

+MODEL_FOR_IMAGE_SEGMENTATION_MAPPING_NAMES = OrderedDict(
+    [
+        # Model for Image Segmentation mapping
+        ("detr", "DetrForSegmentation"),


DetrForSegmentation was probably not the best name. Could we add an alias called DetrForImageSegmentation? Or should we distinguish between the different kinds of segmentation (this particular one is panoptic segmentation)

If I'm not mistaken, as noted in this discussion, we will not make any distinction between different segmentation subtasks and expect models/architectures to implement post_process_segmentation method that implements "the most detailed version of subtask" they can.

More details:
This output shape

List(Dict( mask: str, // base64 str label: float, score: int, parent: int // -1 for all for DetrForSegmentation ))

will allow us to implement part-aware panoptic segmentation (which is the most detailed segmentation subtask). And using this output, a user can re-construct semantic, instance or any othre segmentation subtask if they want

I like the suggestion of DetrForImageSegmentation. Do you want me to add this alias in a different PR?

Yes, adding the alias in a different PR sounds good to me.

src/transformers/pipelines/image_segmentation.py

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

Narsil

LGTM !

mishig25 · 2021-10-08T07:38:37Z

Please let me know if I should merge this PR @Narsil @NielsRogge @LysandreJik

Narsil · 2021-10-08T07:41:20Z

It's gtg for me.

* Implement img seg pipeline * Update src/transformers/pipelines/image_segmentation.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/pipelines/image_segmentation.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update output shape with individual masks * Rm dev change * Remove loops in test Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

Implement img seg pipeline

dd5c269

osanseviero mentioned this pull request Mar 16, 2022

Tracking integration for Image Segmentation huggingface/hub-docs#43

Closed

6 tasks

mishig25 requested review from NielsRogge and Narsil October 1, 2021 09:44

Narsil reviewed Oct 4, 2021

View reviewed changes

NielsRogge reviewed Oct 4, 2021

View reviewed changes

src/transformers/pipelines/image_segmentation.py Outdated Show resolved Hide resolved

NielsRogge reviewed Oct 4, 2021

View reviewed changes

src/transformers/pipelines/image_segmentation.py Outdated Show resolved Hide resolved

mishig25 and others added 4 commits October 4, 2021 21:18

Update src/transformers/pipelines/image_segmentation.py

f2fa620

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

Update src/transformers/pipelines/image_segmentation.py

87b613d

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

Update output shape with individual masks

c240819

Rm dev change

3ac0387

mishig25 requested review from Narsil and NielsRogge October 6, 2021 10:01

Remove loops in test

0f0f07c

Narsil approved these changes Oct 7, 2021

View reviewed changes

mishig25 requested a review from LysandreJik October 8, 2021 07:37

mishig25 merged commit 026866d into huggingface:master Oct 8, 2021

mishig25 mentioned this pull request Nov 1, 2021

Put load_image function in image_utils.py & fix image rotation issue #14062

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Image Segmentation pipeline #13828

Image Segmentation pipeline #13828

mishig25 commented Oct 1, 2021 •

edited

Loading

Narsil left a comment

Narsil Oct 4, 2021

NielsRogge Oct 4, 2021

mishig25 Oct 5, 2021 •

edited

Loading

mishig25 Oct 8, 2021

NielsRogge Oct 8, 2021

Narsil left a comment

mishig25 commented Oct 8, 2021

Narsil commented Oct 8, 2021

	logger.warning("No subtask was supplied, defaulted to panoptic")
	return self.post_process_panoptic(outputs, processed_sizes, threshold=threshold)

Image Segmentation pipeline #13828

Image Segmentation pipeline #13828

Conversation

mishig25 commented Oct 1, 2021 • edited Loading

What does this PR do?

TLDR

API specifications

Two design choices I've made and would like to discuss (& modify if needed):

Before submitting

Who can review?

Update 1:

Narsil left a comment

Choose a reason for hiding this comment

Narsil Oct 4, 2021

Choose a reason for hiding this comment

NielsRogge Oct 4, 2021

Choose a reason for hiding this comment

mishig25 Oct 5, 2021 • edited Loading

Choose a reason for hiding this comment

mishig25 Oct 8, 2021

Choose a reason for hiding this comment

NielsRogge Oct 8, 2021

Choose a reason for hiding this comment

Narsil left a comment

Choose a reason for hiding this comment

mishig25 commented Oct 8, 2021

Narsil commented Oct 8, 2021

mishig25 commented Oct 1, 2021 •

edited

Loading

mishig25 Oct 5, 2021 •

edited

Loading