Add segment-anything models (SAM) to model zoo

[Rusteam]

What changes are proposed in this pull request?

Add three segment-anything models to the model zoo.

How is this patch tested? If it is not, please explain why.

Added an extra intensive model test for sam backbone.

Release Notes

Is this a user-facing change that should be mentioned in the release notes?

• No. You can skip the rest of this section.
• Yes. Give a description of this change to be included in the release
  notes for FiftyOne users.

Segment-anything model is now available in the model zoo. It can perform automatic mask generation and guided mask generation with prompts.

What areas of FiftyOne does this PR affect?

• App: FiftyOne application changes
• Build: Build and test infrastructure changes
• Core: Core fiftyone Python library changes
• Documentation: FiftyOne documentation changes
• Other

[codecov]

Codecov Report

Patch coverage: 73.29% and project coverage change: -13.63 ⚠️

Comparison is base (94b3988) 62.24% compared to head (cd51f71) 48.61%.

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #3019       +/-   ##
============================================
- Coverage    62.24%   48.61%   -13.63%     
============================================
  Files          250      227       -23     
  Lines        45578    33359    -12219     
  Branches       319      319               
============================================
- Hits         28368    16217    -12151     
+ Misses       17210    17142       -68     

Flag	Coverage Δ
app	48.61% <73.29%> (-0.01%)	⬇️
python	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
app/packages/state/src/hooks/useSetView.ts	18.32% <0.00%> (ø)
app/packages/state/src/routing/RouterContext.ts	41.53% <50.00%> (ø)
app/packages/state/src/recoil/groups.ts	48.59% <58.33%> (ø)
app/packages/state/src/hooks/useSchemaSettings.ts	83.01% <72.95%> (ø)
..._generated__/searchSelectFieldsMutation.graphql.ts	100.00% <100.00%> (ø)
...packages/relay/src/mutations/searchSelectFields.ts	100.00% <100.00%> (ø)

... and 24 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[Rusteam]

@brimoor Need your input on the following issue. I'm following this tutorial to add SAM to the model zoo, so far AMG mode worked fine. In order to implement prompting SAM with points and boxes I want to let users specify keypoint/detection label field while calling dataset.apply_model(..., points_from=<keypoint_label_field>, boxes_from=None) and then pull up points and bounding boxes from a sample collection. However in order to achieve that, I have to parse points and boxes before the model.predict_all() method here and then pass them as parameters to that call. This should be backward-compatible but looks like a major change. What do you think is the best scenario to achieve that?

[timmermansjoy]

also a idea and idk if it would bloat this PR. but if you would integrate groundingDINO first you would also add a parameter of "objects" or so and then just prompt the model for objects.

[Rusteam]

Thanks for the suggestion @timmermansjoy. I am myself not familiar with grounding Dino and that's why I want to limit the scope to segment-anything for the moment. After it is integrated, we can think about adding grounding Dino.

[brimoor]

@Rusteam hmm, I see what you mean. Perhaps there should be a way for a Model instance to declare that it needs the sample objects to be supplied along with the images during inference?

prediction = model.predict(img, sample)
predictions = model.predict_all(imgs, samples)

One stylistic way to achieve this would be to implement a SampleMixin class (named similarly to PromptMixin and EmbeddingsMixin).

Note that this line:

fiftyone/fiftyone/core/models.py

Line 312 in 963418d

samples = samples.select_fields()

will need to be excluded for models that inherit from SampleMixin, as they'll need access to all fields in principle (the select_fields() was added to minimize database read bandwidth, since, until now, inference never required anything beyond the image pixels).

Alternatively there could be a SampleMixin.needs_fields property that returns a list of sample fields that should be included:

if isinstance(model, SampleMixin):
    samples = samples.select_fields(model.needs_fields)
else:
    samples = samples.select_fields()

...

if isinstance(model, SampleMixin):
    predictions_batch = model.predict_all(imgs_batch, samples_batch)
else:
    predictions_batch = model.predict_all(imgs_batch)

[Rusteam]

I've slightly changed the behavior of Keypoints from the @jacobmarks blogpost. In this implementation, each Keypoint object will be processed individually with all points passed as positive labels (and no background labels). Also a user can select which mask (0,1, or 2) to use. I guess this requires a short tutorial, which I can work on after we complete this integration.

[Rusteam]

@brimoor hi, this is now ready for testing and integration.

[Rusteam]

@brimoor hey any updates?

[brimoor]

@Rusteam LGTM (sorry for the delay! 😅)

I'm going to merge this into a feature branch and add a couple additional enhancements. Then we'll merge into develop and it will be included in the next release 📈