ODIN is an open source diagnosis framework for generic ML classification tasks and for CV object detection and instance segmentation tasks that lets developers add meta-annotations to their data sets, compute performance metrics split by meta-annotation values, and visualize diagnosis reports.
ODIN is agnostic to the training platform and input formats and can be extended with application- and domain-specific meta-annotations and metrics with almost no coding. It integrates a rapid annotation tool for classification, object detection and instance segmentation data sets.
We have added support to the Time Series analysis. The documentation will be soon available!
For the complete documentation, please visit the Odin website
Developers can run all metrics, or a subset thereof, for a single class or a set of classes. The values of all metrics are reported using diagrams of multiple types, which can be visualized and saved. It is also possible to compare multiple models on the same analysis.
Metrics | Binary Classification | Single-label Classification | Multi-label Classification | Object Detection | Instance Segmentation | |
---|---|---|---|---|---|---|
Base Metrics | Accuracy | yes | yes | yes | n/a | n/a |
Error Rate | yes | yes | yes | n/a | n/a | |
Precision | yes | yes | yes | yes | yes | |
Recall | yes | yes | yes | yes | yes | |
F1 Score | yes | yes | yes | yes | yes | |
Average Precision | yes | yes | yes | yes | yes | |
Precision-Recall AUC | yes | yes | yes | yes | yes | |
ROC AUC | yes | yes | yes | n/a | n/a | |
F1 AUC | yes | yes | yes | yes | yes | |
Custom Metric | yes | yes | yes | yes | yes | |
Curves | Precision-Recall | yes | yes | yes | yes | yes |
F1 | yes | yes | yes | yes | yes | |
ROC | yes | yes | yes | n/a | n/a | |
CAMs Metrics | Global IoU | yes | yes | yes | n/a | n/a |
Component IoU | yes | yes | yes | n/a | n/a | |
Irrelevant Attention | yes | yes | yes | n/a | n/a | |
Bbox Coverage | yes | yes | yes | n/a | n/a |
Analysis | Binary Classification | Single-label Classification | Multi-label Classification | Object Detection | Instance Segmentation | |
---|---|---|---|---|---|---|
Distribution of Classes | Total | yes | yes | yes | yes | yes |
Per-property | yes | yes | yes | yes | yes | |
Distribution of Properties | Total | yes | yes | yes | yes | yes |
Per-category | yes | yes | yes | yes | yes | |
Co-occurrence Matrix | Total | n/a | n/a | yes | yes | yes |
Analysis | Models Comparison | Binary Classification | Single-label Classification | Multi-label Classification | Object Detection | Instance Segmentation | |
---|---|---|---|---|---|---|---|
Performance Analysis | Per-property | yes | yes | yes | yes | yes | yes |
Sensitivity and Impact Analysis | Per-property | yes | yes | yes | yes | yes | yes |
Distribution of TP | Total | yes | no | yes | yes | yes | yes |
Per-property | yes | yes | yes | yes | yes | yes | |
Distribution of FP | Total | yes | no | yes | yes | yes | yes |
Per-property | yes | yes | yes | yes | n/a | n/a | |
Distribution of FN | Total | yes | no | yes | yes | yes | yes |
Per-property | yes | yes | yes | yes | yes | yes | |
Distribution of TN | Total | yes | no | yes | yes | n/a | n/a |
Per-property | yes | yes | yes | yes | n/a | n/a | |
Confusion Matrix | Total | n/a | yes | yes | yes | n/a | n/a |
Per-category | n/a | no | yes | yes | n/a | n/a | |
Per-property | n/a | yes | yes | yes | n/a | n/a | |
FP Categorization and Impact* | Per-category | yes | no | yes | yes | yes | yes |
FP Trend | Per-category | no | no | yes | yes | yes | yes |
FN Categorization | Per-category | yes | no | yes | yes | yes | yes |
Curve Analysis | Total | yes | yes | yes | yes | yes | yes |
Per-category | yes | no | yes | yes | yes | yes | |
Reliability Analysis | Total | n/a | yes | yes | yes | yes | yes |
Per-category | n/a | no | yes | yes | yes | yes | |
Top-1 Top-5 Analysis | Total | no | n/a | yes | n/a | n/a | n/a |
Per-category | no | n/a | yes | n/a | n/a | n/a | |
IoU Analysis | Per-category | no | n/a | n/a | n/a | yes | yes |
CAMs Analysis | Total | yes | yes | yes | yes | n/a | n/a |
Per-category | no | no | yes | yes | n/a | n/a | |
Performance Summary | Total | yes | yes | yes | yes | yes | yes |
Per-category | yes | no | yes | yes | yes | yes | |
Per-property | yes | yes | yes | yes | yes | yes |
Meta-annotation editing is supported by a Jupyter Notebook that given the ground truth and the meta-annotation values, allows the developer to iterate on the samples and select the appropriate value, which is saved in a standard format and can be analysed with the illustrated diagnosis functions.
In addition to editing meta-annotations, ODIN also supports the creation of a classification, object detection or instance segmentation data set. The annotator can be configured to associate training labels to data samples and to draw bounding boxes or segmentation masks over images and label them. Moreover, at the same step of the association of the training labels, it is also possible to add meta-annotations. The resulting data set is saved in a standard format and can be analysed with the illustrated diagnosis functions.
A GUI realized as a Jupyter Notebook enables the inspection of the data set and model predictions. The visualization can be executed:
- on all the samples
- limited to the samples of a class
- limited to the samples with a certain meta-annotation value
- limited to the samples of a class with a given meta-annotation value
- limited to the true positive predicted samples (of all the classes or for the specific ones)
- limited to the false positive predicted samples (of all the classes or for the specific ones)
- limited to the false negative predicted samples (of all the classes or for the specific ones)
- limited to a specific error type of the false positive predicted samples (of all the classes or for the specific ones)
The Visualizer allows also the visualization of the Class Activation Maps.
For the analysis proposed, we need mainly two inputs: (1) the ground truth containing the observations of the dataset, with their corresponding classes, meta-properties and bbox or segmentation mask when applies (can be generated with the previously described annotators) (2) predictions of the model to evaluate.
For the CAMs analysis, in addition to the ground truth file we need also the Class Activation Maps.
Examples are found in test-data.
For localization, we use COCO format (for Pascal VOC an example of parser is provided). For classification an adaptation of COCO is employed:
{
"categories" : [{"id":1, "name":"cat1"}, {"id":2, "name":"cat2"}],
"observations": [ --> # in COCO this is images we renamed it to accept non-images datasets
{
"id": 1,
"file_name": "example.png",
"category": 1, #--> for single label,
"categories": [1, 2] #--> for multi-label
"...": "..", # any other annotation
"...": ".."
}
]
}
For the CAMs analysis, the classification file must contain also the bounding box/segmentation mask annotations:
{
"categories" : [{"id":1, "name":"cat1"}, {"id":2, "name":"cat2"}],
"observations": [
{
"id": 1,
"file_name": "example.png",
"category": 1, #--> for single label,
"categories": [1, 2] #--> for multi-label
"...": "..", # any other annotation
"...": ".."
}
]
"annotations": [
{
"id": 1,
"image_id": 1,
"bbox": [
50,
75,
483,
1323
]
"category_id": 2
}
]
}
One .txt for category (categoryName.txt) with one line for each prediction in the following format
For bounding boxes:
image_id, confidence_score, min_x, min_y, width, height
For segmentation masks:
image_id, confidence_score, x1, y1, x2, y2, ..., xn, yn
For classification:
observation_id, confidence_score
Additionally, record["file_name"] can be used instead of ID for matching the predictions to the GT, providing a parameter indicator "match_on_filename=True", when creating the dataset.
One .npy for image (file_name.npy) of size h x w x c where h and w are the height and width of the image and c is the number of categories.
Sometimes the classes names or properties values are too long and make the graphs less readable. When an Analyzer class is instantiated (using the classes of the examples), a json file (by default called properties.json but can be customized) is created. Here we can modify the "display names" of the different attributes.
The folder "examples" contains notebooks that go through the main functionalities (Analyzer, Annotator, Visualizer), using the dataset ARTDL (http://www.artdl.org/). Some examples are provided using RacePlanes(https://www.cosmiqworks.org/RarePlanes/) or SpamSMS (https://www.dt.fee.unicamp.br/~tiago/smsspamcollection/). Other notebooks are for you to play with your own dataset. We tried to make the functions easy to use and intuitive, as can be seen in the examples.
The framework and its requirements can be install with
pip install -e . --user
After this command, the framework appears as a pip package and can be used everywhere as
from odin import ....
Every change is reflected in the pip package without requiring any additional installation step.
N.B. To be able to use the meta-annotation extractor, two more packages need to be installed (tensorflow and mtcnn). Simply execute the following command:
pip install tensorflow mtcnn
We provide the Dockerfile to create a docker image.
docker build -t odin .
Once the 'odin' image has been created, just run:
docker run -it -p 8888:8888 odin
If you use ODIN or wish to refer it, please use the following BibTex entry.
@inproceedings{torres2020odin,
title={ODIN: An Object Detection and Instance Segmentation Diagnosis Framework},
author={Torres, Rocio Nahime and Fraternali, Piero and Romero, Jesus},
booktitle={European Conference on Computer Vision},
pages={19--31},
year={2020},
organization={Springer}
}
@inproceedings{torres2021odin,
title={ODIN: Pluggable Meta-annotations and Metrics for the Diagnosis of Classification and Localization},
author={Torres, Rocio Nahime and Milani, Federico and Fraternali, Piero},
booktitle={International Conference on Machine Learning, Optimization, and Data Science},
pages={383--398},
year={2021},
organization={Springer}
}
@article{zangrando2022odin,
title={ODIN TS: A Tool for the Black-Box Evaluation of Time Series Analytics},
author={Zangrando, Niccol{\`o} and Torres, Rocio Nahime and Milani, Federico and Fraternali, Piero},
journal={Engineering Proceedings},
volume={18},
number={1},
pages={4},
year={2022},
publisher={MDPI}
}