Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLIP to the model zoo #1691

Merged
merged 6 commits into from Sep 8, 2022
Merged

Add CLIP to the model zoo #1691

merged 6 commits into from Sep 8, 2022

Conversation

Rusteam
Copy link
Contributor

@Rusteam Rusteam commented Apr 21, 2022

What changes are proposed in this pull request?

Adds:

  • a CLIP ViT-Base-32 model to the model zoo. It's available for classification and embeddings.
  • zero-shot model zoo tag to let users pass custom class labels to classification. Usage template is updated to include an example of passing custom class labels. (by default it will use voc_labels.txt)
  • Simple text tokenizer required for CLIP's text encoder.

How is this patch tested? If it is not, please explain why.

Added an extra unit-test for zero-shot label predictions.

Release Notes

Is this a user-facing change that should be mentioned in the release notes?

  • No. You can skip the rest of this section.
  • Yes. Give a description of this change to be included in the release
    notes for FiftyOne users.

Make a zero-shot class label predictions using CLIP model by using default labels or passing your own labels.

What areas of FiftyOne does this PR affect?

  • App: FiftyOne application changes
  • Build: Build and test infrastructure changes
  • Core: Core fiftyone Python library changes
  • Documentation: FiftyOne documentation changes
  • Other

@Rusteam Rusteam changed the title [zoo] add CLIP to the model zoo WIP: [zoo] add CLIP to the model zoo Apr 21, 2022
@brimoor brimoor requested a review from a team April 21, 2022 19:05
@brimoor brimoor self-assigned this Apr 21, 2022
@brimoor brimoor added the feature Work on a feature request label Apr 21, 2022
@Rusteam Rusteam changed the title WIP: [zoo] add CLIP to the model zoo [zoo] add CLIP to the model zoo Apr 22, 2022
Copy link
Contributor

@brimoor brimoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work, thanks for the contribution! 💯 🥇

Just left a couple formatting requests for you

fiftyone/utils/clip.py Outdated Show resolved Hide resolved
fiftyone/utils/clip.py Outdated Show resolved Hide resolved
fiftyone/zoo/models/torch.py Outdated Show resolved Hide resolved
fiftyone/zoo/models/manifest-torch.json Show resolved Hide resolved
tests/intensive/model_zoo_tests.py Show resolved Hide resolved
docs/scripts/make_model_zoo_docs.py Show resolved Hide resolved
@Rusteam
Copy link
Contributor Author

Rusteam commented Apr 25, 2022

Awesome work, thanks for the contribution! 💯 🥇

Just left a couple formatting requests for you

Thanks it was a bit challenging. I guess because it's intertwined with the eta package and the bulk of the code is based upon config parsers. Overall, it was an interesting experience.

Hope it will be useful for the community.

@brimoor brimoor changed the title [zoo] add CLIP to the model zoo Add CLIP to the model zoo May 13, 2022
@Rusteam
Copy link
Contributor Author

Rusteam commented Sep 8, 2022

Hi there. Can we get this one merged? I can rebase this branch to v0.16.5

@brimoor
Copy link
Contributor

brimoor commented Sep 8, 2022

Hey @Rusteam absolutely, merging now! This will be included in fiftyone==0.17 being released later today 🎉

Copy link
Contributor

@brimoor brimoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@brimoor brimoor merged commit e60f96a into voxel51:develop Sep 8, 2022
@Rusteam
Copy link
Contributor Author

Rusteam commented Sep 8, 2022

Thanks a lot. Just preparing to use it actually.

@brimoor brimoor mentioned this pull request Sep 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Work on a feature request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants