MetaDetect3D

MetaDetect3D is a post-processing tool for 3D object detection neural networks that improve calibration, performance and can be used for efficient label error detection as well.

Preface

Advances in the development of deep neural networks (NNs) over the last two decades have made it possible for an artificial intelligence (AI) not only to tackle increasingly complex tasks, but even to master them at a human level. Computer vision, natural language processing and reinforcement learning approaches are just some of the few areas covered by the now huge field of AI. Above all, the task of autonomous driving keeps scientists around the world engaged in perfecting its capabilities and reducing its weaknesses, with the goal of making it safer than any human driver. While the two main approaches, visual and LiDAR (light detection and ranging) based, already work well in many traffic scenarios, there are still many cases where the deep learning model makes incorrect predictions. In particular, it often predicts an object where none is (false positive), or it predicts no object where one should have been (false negative). Both of these errors can be very dangerous in practice.

This repository presents the work of my Master Thesis about Meta Detection methods. MetaDetect3D is inspired by MetaDetect created and developed at the University of Wuppertal in Germany.

MetaDetect3D is a post-processing tool for 3D object detection neural networks. The idea behind MetaDetect3D is to use informations provided by an 3D Object Detector, specifically its input, output and from its proposals in order to define features and metrics on those in order to create a new dataset. The new dataset is used to train a new ML model which further refines the previous output of the underlying object detector.

Meta models can have two purposes: The first is to evaluate the predictions by assigning them either a numerical value between 0 and 1 (Meta regression) or a binary label or probability (Meta classification). This functionality is crucial if one does not want to rely solely on the confidence score output by the object detector. If trained properly, this can lead to a better quality estimate of the predictions obtained from the object detector. Second, one can use the Meta detector to spot misplaced or missing labels, or stated differently, to detect label errors. This can be done by assuming that the Meta regressor assigns high values, or equivalently that the Meta classifier assigns a true label to boxes where it assumes the prediction is correct, even though the ground truth does not contain the label. This would yield potentially missing labels. Similarly, the Meta models should assign low values (or a false label) to a prediction where no box is assumed even though a box was predicted. In this case, if the prediction overlaps with a ground truth box, the label may be incorrect.

The graph below displays the workflow of MetaDetect3D in detail.

In words the steps of the MetaDetect3D methods are:

1. Train a object detector on training set (and validate it on validation set)
2. Crawling prediction 3D object detect and labels of validation and test set
3. Computation of Meta dispersion measures on previously crawled data
4. MetaDetect3D training pipeline including parameter search (on validation set)
5. MetaDetect3D evaluation on best Meta models (on test set)

Main Library Features

• High Level API
• Meta Regression and Classification possible
• Simple Post-Processing Tool for any 3D Object Detector
• Various Machine Learning Models available

Table of Contents

• Installation and Setup
• Usage and Examples
• Citing
• License
• References
• Authors and Acknowledgements
• Support

Installation and Setup

To get the repository running just check the following requirements.

Requirements Windows or Linus

1. Python 3.6 or higher
2. torch >= 1.10.1
3. scikit-learn >= 0.24.1
4. numpy >= 1.19.5
5. pandas >= 1.15
6. mmcv-full >= 1.4.6
7. mmcls >= 0.21.0
8. mmdet >= 2.22.0
9. mmdet3d >= 1.0.0rc0
10. mmsegmentation >= 0.22.1

Furthermore just execute the following command to download and install the git repository.

This will not install any open-mmlab package like mmcv, mmdet3d, etc.. This has to be done manually.

Clone Repository

$ git clone https://github.com/JanMarcelKezmann/MetaDetect3D.git

or directly install it:
Pip Install Repository

$ pip install git+https://github.com/JanMarcelKezmann/MetaDetect3D.git

Usage and Examples

Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.

• [Jupyter Notebbok] MetaDetect3D on KITTI Dataset here

Available ML Models

CLASSIFICATION_MODELS = ['logistic', 'ridge_cls', 'random_forest_cls', 'gb_cls', "mlp_cls"] REGRESSION_MODELS = ['ridge_reg', 'lasso_reg', 'random_forest_reg', 'gb_reg', "mlp_reg"]

Model	Classifier	Regressors
Logistic	✔️
Ridge	✔️	✔️
Random Forest	✔️	✔️
Gradient Boosting	✔️	✔️
MLP	✔️	✔️

Object Detectors

• PointPillars
• CenterPoint

Predefined Dataset Pipelines

• KITTI
• NuScenes
• (APTIV) custom dataset

This list should help you for adapting the code on custom datasets on your own.

Citing

@misc{Kezmann:2022,
  Author = {Jan-Marcel Kezmann},
  Title = {MetaDetect3D},
  Year = {2022},
  Publisher = {GitHub},
  Journal = {GitHub repository},
  Howpublished = {\url{https://github.com/JanMarcelKezmann/MetaDetect3D}}
} 

License

Project is distributed under MIT License.

References

Thank you for all the papers and people that made this repository possible.

Authors and Acknowledgment

Support

Feel free to raise an Issue in case something does not work properly.