This plugin allows you to perform zero-shot prediction on your dataset for the following tasks:
- Image Classification
- Object Detection
- Instance Segmentation
- Semantic Segmentation
Given a list of label classes, which you can input either manually, separated by commas, or by uploading a text file, the plugin will perform zero-shot prediction on your dataset for the specified task and add the results to the dataset under a new field, which you can specify.
- 2023-10-27: Added support for MetaCLIP for image classification
- 2023-10-20: Added support for AltCLIP and Align for image classification and GroupViT for semantic segmentation
As a starting point, this plugin comes with at least one zero-shot model per task. These are:
- Image Classification: CLIP, AltCLIP, MetaCLIP, and Align
- Object Detection: Owl-ViT
- Instance Segmentation: Owl-ViT + Segment Anything (SAM)
- Semantic Segmentation: CLIPSeg and GroupViT
Most of the models used are from the HuggingFace Transformers library, and CLIP and SAM models are from the FiftyOne Model Zoo
Note— For SAM you will need to have Facebook's segment-anything
library installed.
You can see the implementations for all of these models in the following files:
classification.py
detection.py
instance_segmentation.py
semantic_segmentation.py
These models are "registered" via dictionaries in each file. In classification.py
, for example, the dictionary is:
CLASSIFICATION_MODELS = {
"CLIP": {
"activator": CLIP_activator,
"model": CLIPZeroShotModel,
"name": "CLIP",
},
"AltCLIP": {
"activator": AltCLIP_activator,
"model": AltCLIPZeroShotModel,
"name": "AltCLIP",
},
"MetaCLIP-H14": {
"activator": MetaCLIP_activator,
"model": MetaCLIPZeroShotModel,
"name": "MetaCLIP-H14",
},
"Align": {
"activator": Align_activator,
"model": AlignZeroShotModel,
"name": "Align",
},
}
The activator
checks the environment to see if the model is available, and the model
is a fiftyone.core.models.Model
object that is instantiated with the model name and the task. The name
is the name of the model that will be displayed in the dropdown menu in the plugin.
If you want to add your own model, you can add it to the dictionary in the corresponding file. For example, if you want to add a new image classification model, you can add it to the CLASSIFICATION_MODELS
dictionary in classification.py
:
CLASSIFICATION_MODELS = {
"CLIP": {
"activator": CLIP_activator,
"model": CLIPZeroShotModel,
"name": "CLIP",
},
..., # other models
"My Model": {
"activator": my_model_activator,
"model": my_model,
"name": "My Model",
}
}
💡 You need to implement the activator
and model
functions for your model. The activator
should check the environment to see if the model is available, and the model
should be a fiftyone.core.models.Model
object that is instantiated with the model name and the task.
fiftyone plugins download https://github.com/jacobmarks/zero-shot-prediction-plugin
If you want to use AltCLIP, Align, Owl-ViT, CLIPSeg, or GroupViT, you will also need to install the transformers
library:
pip install transformers
If you want to use SAM, you will also need to install the segment-anything
library:
pip install git+https://github.com/facebookresearch/segment-anything.git
All of the operators in this plugin can be run in delegated execution mode. This means that instead of waiting for the operator to finish, you schedule the operation to be performed separately. This is useful for long-running operations, such as performing inference on a large dataset.
Once you have pressed the Schedule
button for the operator, you will be able to see the job from the command line using FiftyOne's command line interface:
fiftyone delegated list
will show you the status of all delegated operations.
To launch a service which runs the operation, as well as any other delegated operations that have been scheduled, run:
fiftyone delegated launch
Once the operation has completed, you can view the results in the App (upon refresh).
After the operation completes, you can also clean up your list of delegated operations by running:
fiftyone delegated cleanup -s COMPLETED
- Select the task you want to perform zero-shot prediction on (image classification, object detection, instance segmentation, or semantic segmentation), and the field you want to add the results to.
- Perform zero-shot image classification on your dataset
- Perform zero-shot object detection on your dataset
- Perform zero-shot instance segmentation on your dataset
- Perform zero-shot semantic segmentation on your dataset