add support for rgb overlays #2483

sashankaryal · 2023-01-04T17:57:05Z

Resolves #1730.

Change log

Adds support for visualizing RGB segmentations in the App
Adds support for RGB segmentations in evaluate_segmentations()
Adds support for passing RGB mask targets to objects_to_segmentations()
Adds a new transform_segmentations() utility to transform existing masks between grayscale <> RGB

Example usage

import fiftyone as fo
import fiftyone.zoo as foz
import fiftyone.utils.labels as foul

mask_targets = {"#FF6D04": "person", "#499CEF": "cat", "#6D04FF": "dog"}

dataset = foz.load_zoo_dataset(
    "coco-2017",
    split="validation",
    label_types=["segmentations"],
    classes=["person", "cat", "dog"],
    label_field="instances",
    max_samples=25,
    only_matching=True,
)

foul.objects_to_segmentations(
    dataset,
    "instances",
    "segmentations",
    mask_targets=mask_targets,
)

dataset.default_mask_targets = mask_targets

# Test in-database RGB segmentations in App
session = fo.launch_app(dataset)

foul.export_segmentations(
    dataset, "segmentations", "/tmp/segmentations", overwrite=True
)

# Test on-disk RGB segmentations in App
session.refresh()

# Test evaluating RGB segmentations
dataset.rename_sample_field("segmentations", "ground_truth")
dataset.clone_sample_field("ground_truth", "predictions")

results1 = dataset.evaluate_segmentations(
    "predictions", gt_field="ground_truth", mask_targets=mask_targets
)
results1.print_report()

results2 = dataset.evaluate_segmentations("predictions", gt_field="ground_truth")
results2.print_report()

# Convert to grayscale
targets_map = {"#FF6D04": 1, "#499CEF": 2, "#6D04FF": 3}
foul.import_segmentations(dataset, "ground_truth")
foul.transform_segmentations(dataset, "ground_truth", targets_map)
foul.transform_segmentations(
    dataset, "predictions", targets_map, update_mask_targets=True
)

# Edit grayscale values
foul.transform_segmentations(
    dataset, "predictions", {1: 70, 2: 140, 3: 210}, update_mask_targets=True
)

# Convert back to RGB
foul.transform_segmentations(
    dataset, "predictions", {70: "#FF6D04", 140: "#499CEF", 210: "#6D04FF"}
)

codecov · 2023-01-04T18:03:56Z

Codecov Report

Base: 48.18% // Head: 61.73% // Increases project coverage by +13.55% 🎉

Coverage data is based on head (8130059) compared to base (f4e5c87).
Patch coverage: 64.30% of modified lines in pull request are covered.

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #2483       +/-   ##
============================================
+ Coverage    48.18%   61.73%   +13.55%     
============================================
  Files          186      207       +21     
  Lines        29099    39884    +10785     
  Branches       240      242        +2     
============================================
+ Hits         14020    24623    +10603     
- Misses       15079    15261      +182

Flag	Coverage Δ
app	`48.18% <58.17%> (+<0.01%)`	⬆️
python	`99.39% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
app/packages/state/src/utils.ts	`21.27% <12.50%> (-1.04%)`	⬇️
app/packages/looker/src/overlays/segmentation.ts	`25.24% <17.27%> (-3.65%)`	⬇️
app/packages/utilities/src/color.ts	`25.22% <18.18%> (-0.76%)`	⬇️
app/packages/state/src/recoil/selectors.ts	`49.87% <23.07%> (-1.26%)`	⬇️
app/packages/looker/src/index.ts	`21.17% <33.33%> (-0.12%)`	⬇️
app/packages/looker/src/numpy.ts	`54.59% <42.85%> (-0.06%)`	⬇️
app/packages/looker/src/overlays/util.ts	`26.43% <50.00%> (+3.46%)`	⬆️
app/packages/looker/src/overlays/detection.ts	`20.66% <100.00%> (ø)`
app/packages/looker/src/overlays/heatmap.ts	`29.91% <100.00%> (ø)`
app/packages/looker/src/state.ts	`98.88% <100.00%> (+0.03%)`	⬆️
... and 27 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

brimoor

@sashankaryal how's this PR coming? A couple things I noticed when testing just now:

The tooltip does not render correctly when viewing a dataset without mask targets explicitly defined in the App (does not consistency appear; and when it does seems to be showing the wrong data?)

I also noticed that pixel values that do not appear in the provided mask_targets are currently being rendered. Is that intentional? Or are you planning to switch to the currently documented behavior that pixels not in masks_targets are not rendered? After testing just now and experiencing the current behavior, I'm thinking it could be fine to go ahead and always render all non-zero classes and just show pixel value when a label is not available.

I can see value both ways:

If only mask_targets are painted, it would give a simple way to change what is rendered by editing mask_targets
If all non-zero values are painted, then there is never any confusion why user's data is not appearing

sashankaryal · 2023-01-09T17:57:47Z

@sashankaryal how's this PR coming? A couple things I noticed when testing just now:

The tooltip does not render correctly when viewing a dataset without mask targets explicitly defined in the App (does not consistency appear; and when it does seems to be showing the wrong data?)

I also noticed that pixel values that do not appear in the provided mask_targets are currently being rendered. Is that intentional? Or are you planning to switch to the currently documented behavior that pixels not in masks_targets are not rendered? After testing just now and experiencing the current behavior, I'm thinking it could be fine to go ahead and always render all non-zero classes and just show pixel value when a label is not available.

I can see value both ways:

If only mask_targets are painted, it would give a simple way to change what is rendered by editing mask_targets

If all non-zero values are painted, then there is never any confusion why user's data is not appearing

Addressed in the new commits

benjaminpkane

Generally the code looks good. Nice job!

app/packages/looker/src/overlays/segmentation.ts

benjaminpkane · 2023-01-11T16:42:02Z

app/packages/looker/src/overlays/util.ts

+    throw new Error("mask targets is invalid");
+  }
+
+  return Object.keys(maskTargets)[0]?.startsWith("#") === true;


This will run quite often, could we cache this somehow?

benjaminpkane · 2023-01-11T16:42:55Z

app/packages/looker/src/worker.ts

-  DETECTIONS,
-]);
+// defined as labels that can have on-disk overlays
+const DENSE_LABELS = new Set([SEGMENTATION, HEATMAP, DETECTION, DETECTIONS]);


Maybe define in @fiftyone/utilities for reuse?

benjaminpkane · 2023-01-11T16:50:39Z

app/packages/looker/src/worker.ts

+        const r = targets[i * mapData.channels];
+        const g = targets[i * mapData.channels + 1];
+        const b = targets[i * mapData.channels + 2];


There is a fair bit of shared complexity in theses functions, like these for loops. You might be able to share some code to improve readability

benjaminpkane · 2023-01-11T17:40:27Z

app/packages/looker/src/overlays/segmentation.ts

+
+      if (!isRgbMaskTargets(maskTargets)) {
+        // getting color here is computationally inefficient, return no color for this edge case
+        return rgbSegmentationInfoWithoutColor;


Can you elaborate here? I'm not sure I understand

This is for the edge case when the mask is multi-channel (RGB) but the mask targets is not RGB (or is missing). In this scenario, we render the RGB mask as is but the tooltip will behave slightly differently in two respects:

The ribbon color won't reflect the segmentation color, it'll be static in the entire canvas (a shade of white, I think).

We won't display any "pixel" info.

This comment is re: (1); while it's possible to get the color of the underlying mask, it involves parsing the RGB buffer. In the happy path case, the RGB buffer is reduced into a targets lookup cable and we don't have this problem.

brimoor

LGTM

brimoor requested review from benjaminpkane and brimoor January 5, 2023 15:04

brimoor assigned sashankaryal Jan 5, 2023

brimoor added the feature Work on a feature request label Jan 5, 2023

sashankaryal marked this pull request as ready for review January 5, 2023 15:44

sashankaryal force-pushed the feature/rgb-mask branch from 3df4a7c to 6c1f6e2 Compare January 6, 2023 00:14

brimoor requested changes Jan 7, 2023

View reviewed changes

sashankaryal force-pushed the feature/rgb-mask branch from 3b07f77 to 588c93f Compare January 9, 2023 17:57

benjaminpkane reviewed Jan 11, 2023

View reviewed changes

sashankaryal and others added 19 commits January 11, 2023 16:21

update data model for rgb mask targets

171071d

add frontend support for rgb overlays

ecc4e96

add tests for rgb mask targets

4b908d9

linting

1a7d9d4

must explicitly declare field type

a15ac03

fix circular import

bf8766f

update validation()

683ee3c

linting

5452c5c

adding support for evaluating and transforming RGB segmentations

10accf3

updating documentation

d1cb92b

update docstring

1bb3baf

handle edge cases

eebb6c3

minor refactoring: let to const

3cc37fe

fix failing test by adding boolean guard for undefined mask targets

97d7f7a

updating docstrings

2acdf28

linting

a229d77

adding unit tests for RGB segementation eval

9a105cb

fix more edge cases

83fd83c

fix bugs; address pr comments

8130059

sashankaryal force-pushed the feature/rgb-mask branch from 588c93f to 8130059 Compare January 13, 2023 05:13

brimoor approved these changes Jan 13, 2023

View reviewed changes

brimoor merged commit 77adcef into develop Jan 13, 2023

brimoor deleted the feature/rgb-mask branch January 13, 2023 12:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for rgb overlays #2483

add support for rgb overlays #2483

sashankaryal commented Jan 4, 2023 •

edited by brimoor

codecov bot commented Jan 4, 2023 •

edited

brimoor left a comment •

edited

sashankaryal commented Jan 9, 2023

benjaminpkane left a comment

benjaminpkane Jan 11, 2023

benjaminpkane Jan 11, 2023

benjaminpkane Jan 11, 2023

benjaminpkane Jan 11, 2023

sashankaryal Jan 13, 2023

brimoor left a comment

add support for rgb overlays #2483

add support for rgb overlays #2483

Conversation

sashankaryal commented Jan 4, 2023 • edited by brimoor

Change log

Example usage

codecov bot commented Jan 4, 2023 • edited

Codecov Report

brimoor left a comment • edited

Choose a reason for hiding this comment

sashankaryal commented Jan 9, 2023

benjaminpkane left a comment

Choose a reason for hiding this comment

benjaminpkane Jan 11, 2023

Choose a reason for hiding this comment

benjaminpkane Jan 11, 2023

Choose a reason for hiding this comment

benjaminpkane Jan 11, 2023

Choose a reason for hiding this comment

benjaminpkane Jan 11, 2023

Choose a reason for hiding this comment

sashankaryal Jan 13, 2023

Choose a reason for hiding this comment

brimoor left a comment

Choose a reason for hiding this comment

sashankaryal commented Jan 4, 2023 •

edited by brimoor

codecov bot commented Jan 4, 2023 •

edited

brimoor left a comment •

edited