Set masks to zero where masks overlap #8213

sam-watts · 2024-01-16T13:44:43Z

No description provided.

…in draw_segmentation_masks

pytorch-bot · 2024-01-16T13:44:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/8213

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (32 Unrelated Failures)

As of commit 967fd1f with merge base 660868b ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

CMake / linux (linux.12xlarge, cpu) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
CMake / linux (linux.g5.4xlarge.nvidia.gpu, cuda, 11.8) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
CMake / macos (macos-12) / macos-job (gh)
The process '/usr/local/bin/git' failed with exit code 128
CMake / macos (macos-m1-12) / macos-job (gh)
The process '/usr/bin/git' failed with exit code 128
CMake / windows (windows.4xlarge, cpu) / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128
CMake / windows (windows.g5.4xlarge.nvidia.gpu, cuda, 11.8) / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128
Docs / build / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Lint / c-source / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Lint / python-source-and-configs / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Lint / python-types / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Prototype tests on Linux / unittests-prototype (3.10, linux.12xlarge, cpu) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Prototype tests on Linux / unittests-prototype (3.11, linux.12xlarge, cpu) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Prototype tests on Linux / unittests-prototype (3.8, linux.12xlarge, cpu) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Prototype tests on Linux / unittests-prototype (3.8, linux.g5.4xlarge.nvidia.gpu, cuda, 11.8) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Prototype tests on Linux / unittests-prototype (3.9, linux.12xlarge, cpu) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Tests / onnx / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Tests / unittests-extended / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Tests / unittests-linux (3.10, linux.12xlarge, cpu) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Tests / unittests-linux (3.11, linux.12xlarge, cpu) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Tests / unittests-linux (3.8, linux.12xlarge, cpu) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Tests / unittests-linux (3.8, linux.g5.4xlarge.nvidia.gpu, cuda, 11.8) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Tests / unittests-linux (3.9, linux.12xlarge, cpu) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Tests / unittests-macos (3.10, macos-12) / macos-job (gh)
The process '/usr/local/bin/git' failed with exit code 128
Tests / unittests-macos (3.11, macos-12) / macos-job (gh)
The process '/usr/local/bin/git' failed with exit code 128
Tests / unittests-macos (3.8, macos-12) / macos-job (gh)
The process '/usr/local/bin/git' failed with exit code 128
Tests / unittests-macos (3.8, macos-m1-12) / macos-job (gh)
The process '/usr/bin/git' failed with exit code 128
Tests / unittests-macos (3.9, macos-12) / macos-job (gh)
The process '/usr/local/bin/git' failed with exit code 128
Tests / unittests-windows (3.10, windows.4xlarge, cpu) / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128
Tests / unittests-windows (3.11, windows.4xlarge, cpu) / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128
Tests / unittests-windows (3.8, windows.4xlarge, cpu) / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128
Tests / unittests-windows (3.8, windows.g5.4xlarge.nvidia.gpu, cuda, 11.8) / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128
Tests / unittests-windows (3.9, windows.4xlarge, cpu) / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2024-01-16T13:44:49Z

Hi @sam-watts!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

sam-watts · 2024-01-16T14:11:04Z

In cases where a user might want to display overlapping segmentation masks (eg. ground truth vs. prediction) the current function is not suitable as it overwrites colours, meaning that masks processed first are not fully visible:

A simple fix is to use iterative multiplicative blending, which shows decent results even in the presence of multiple overlapping masks:

code for examples:

def show(imgs):
    if not isinstance(imgs, list):
        imgs = [imgs]
    fig, axs = plt.subplots(ncols=len(imgs), squeeze=False)
    for i, img in enumerate(imgs):
        img = img.detach()
        img = F.to_pil_image(img)
        axs[0, i].imshow(np.asarray(img))
        axs[0, i].set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])

dims = 5

masks = torch.zeros(dims,256,256).bool()
img = torch.ones(3,256,256).to(torch.uint8) * 255

for i in range(dims):
    center = torch.randint(0, 256, (2,))
    size = torch.randint(20, 50, (2,))
    masks[i, center[0]-size[0]:center[0]+size[0], center[1]-size[1]:center[1]+size[1]] = True

colors = ["red", "green", "blue", "orange", "yellow"]
show(draw_segmentation_masks(img, masks, colors=colors)) # original function
show(_draw_segmentation_masks(img, masks, colors=colors)) # function rewritten in this PR

facebook-github-bot · 2024-01-16T15:08:34Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

sam-watts · 2024-01-17T10:52:48Z

Outstanding question - do we want to include extra logic in tests.test_utils.test_draw_segmentation_masks to check that the multiplicative blending results in lower pixel values in cases where masks overlap?

NicolasHug · 2024-01-17T11:48:02Z

Thanks a lot for this nice addition @sam-watts . The code LGTM.

do we want to include extra logic in tests.test_utils.test_draw_segmentation_masks to check that the multiplicative blending results in lower pixel values in cases where masks overlap?

Yes please! In order to keep it simple, I guess we would have to stop randomly generate the test masks, and instead manually generate 2 masks. 2 overlapping squares should do the job?

…n into seg-masks-mult-blending

sam-watts · 2024-01-17T13:50:39Z

FYI @NicolasHug I simplified the code, as it was not working in certain cases. Instead of blending masks we now set areas to zero where there is more than one mask. This has the benefit of also working well when uint8 images are used instead of [0-1] floats

Tests now added

New version looks like this:

NicolasHug

Thanks a lot @sam-watts ! Just one question about the addition of pillow to the dev dependencies, but otherwise LGTM!

NicolasHug · 2024-01-19T13:46:32Z

CONTRIBUTING.md

@@ -74,7 +74,7 @@ We don't officially support building from source using `pip`, but _if_ you do, y
 #### Other development dependencies (some of these are needed to run tests):

 ```
-pip install expecttest flake8 typing mypy pytest pytest-mock scipy
+pip install expecttest flake8 typing mypy pytest pytest-mock scipy pillow


That shouldn't be needed here as it's already a hard-dependency of torchvision, so it should get installed when you install torchvision from source.

Curious if you encounter any issue here?

hmmm odd. I remember having to install it manually after building from source, as I got a package not installed error from the tests. I'm not familiar with the internals of python setup.py develop, but I can say that I've used pip install -e . instead before. There is some discussion on SO about dependencies not being handled correctly with the former, but I'm not sure how much weight to give that.

Happy to remove this though, I may have just done something odd / could be a personal configuration issue

Thanks for the info - let me just remove this so we can move forward with this PR and if we're getting more reports about this I'll investigate. Thank you!

CONTRIBUTING.md

Reviewed By: vmoens Differential Revision: D55062792 fbshipit-source-id: e76e7e5596a619239cf3e01f9fbc7a7d0002504c Co-authored-by: Nicolas Hug <contact@nicolas-hug.com> Co-authored-by: Nicolas Hug <nh.nicolas.hug@gmail.com>

add multiplicative blending to make overlapping masks display better …

ada9a51

…in draw_segmentation_masks

facebook-github-bot added the cla signed label Jan 16, 2024

update test comment

c86143d

sam-watts marked this pull request as ready for review January 17, 2024 10:48

Merge branch 'main' into seg-masks-mult-blending

e8f4b1d

sam-watts added 4 commits January 17, 2024 13:30

simplify change, alter tests

f1c47a9

add extra assert for overlap region

c4312bd

add pillow to test requirements

9bfd72d

:Merge branch 'seg-masks-mult-blending' of github.com:sam-watts/visio…

f8fce10

…n into seg-masks-mult-blending

sam-watts changed the title ~~add multiplicative blending to make overlapping masks display better …~~ Set masks to zero where masks overlap Jan 17, 2024

NicolasHug approved these changes Jan 19, 2024

View reviewed changes

NicolasHug reviewed Jan 19, 2024

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

NicolasHug and others added 2 commits January 19, 2024 14:21

Update CONTRIBUTING.md

cd219ef

Merge branch 'main' into seg-masks-mult-blending

967fd1f

NicolasHug merged commit 6f0deb9 into pytorch:main Jan 19, 2024
2 of 14 checks passed

sam-watts deleted the seg-masks-mult-blending branch January 19, 2024 14:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set masks to zero where masks overlap #8213

Set masks to zero where masks overlap #8213

sam-watts commented Jan 16, 2024 •

edited

pytorch-bot bot commented Jan 16, 2024 •

edited

facebook-github-bot commented Jan 16, 2024

sam-watts commented Jan 16, 2024 •

edited

facebook-github-bot commented Jan 16, 2024

sam-watts commented Jan 17, 2024

NicolasHug commented Jan 17, 2024

sam-watts commented Jan 17, 2024 •

edited

NicolasHug left a comment

NicolasHug Jan 19, 2024

sam-watts Jan 19, 2024

NicolasHug Jan 19, 2024

Set masks to zero where masks overlap #8213

Set masks to zero where masks overlap #8213

Conversation

sam-watts commented Jan 16, 2024 • edited

pytorch-bot bot commented Jan 16, 2024 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/8213

✅ You can merge normally! (32 Unrelated Failures)

facebook-github-bot commented Jan 16, 2024

Action Required

Process

sam-watts commented Jan 16, 2024 • edited

facebook-github-bot commented Jan 16, 2024

sam-watts commented Jan 17, 2024

NicolasHug commented Jan 17, 2024

sam-watts commented Jan 17, 2024 • edited

NicolasHug left a comment

Choose a reason for hiding this comment

NicolasHug Jan 19, 2024

Choose a reason for hiding this comment

sam-watts Jan 19, 2024

Choose a reason for hiding this comment

NicolasHug Jan 19, 2024

Choose a reason for hiding this comment

sam-watts commented Jan 16, 2024 •

edited

pytorch-bot bot commented Jan 16, 2024 •

edited

sam-watts commented Jan 16, 2024 •

edited

sam-watts commented Jan 17, 2024 •

edited