Related to our paper Unified Object Detector for Different Modalities based on Vision Transformers.
Our unified model can process RGB images, pseudo images converted from point clouds or inter-modality mixing of RGB image and pseudo images converted from point clouds.
Comparison of other systems can be seen 
This repo contains the supported code and configuration files to reproduce object detection results of [simCrossTrans]. It is modified based on Swin Transformer for object detection. Original Readme
Those are the base model used for the UODDM work. You can download them from the official SWIN transformer repo, you can also download a backup from the link(Google Drive) provided here:
| Finetune dataset | model | checkpoint |
|---|---|---|
| COCO | Swin-T | swin-t-model |
| COCO | Swin-S | swin-s-model |
The UODDM work was finetuning the above model based on SUN RGBD dataset. It has two models based on different modalities:
- INPUT A: RGB.
- INPUT B: RGB and DHS and RGB DHS mixed based on chessboard mixture.
We also had a input only as DHS model can be found in the simCrossTrans work. Here the performance based on mAP50 for SUNRGBD10, which includes a 10 common categories. Details please check the UODDM paper.
| Finetune dataset | input | model | checkpoint | configure file | performance on RGB validation | performance on DHS validation | performance when both RGB and DHS are available | log |
|---|---|---|---|---|---|---|---|---|
| SUN RGBD | INPUT A | swin-t | basedRGB | cfg | 53.9 | N/A | N/A | log |
| SUN RGBD | INPUT B | swin-t | basedRGBandDHSandRGBDHSmixed | cfg | 54.2 | 55.8 | 58.1 | test_on_RGB test on RGB DHS mixed |
The sun rgbd dataset training and test can be found in the sunrgbd folder, if you want to train the sunrgbd dataset based on pretrained model on COCO, please do the following:
cd sunrgbd
./shell_script/uoddm/train_swin_transform.shYou need download a pretrained model from the COCO dataset and you can find the models in the MODEL session. If you want to train a RGB image, please use:# train RGB with pretrained weights from coco for 100 epochs"
The SUN RGBD dataset also has 80 categories to align with COCO dataset. The SUNRGBD is direclty overwritten the COCO dataset's class, see this line If you want to directly use the pretrained model from SUN RGBD dataset, you need use the following customized mmdetection (updating the categories name to SUN RGBD and add some inference code).
https://github.com/liketheflower/mmdetection_betaRun the following shell script:
./sunrgbd/shell_script/uoddm/inference/inference.shPlease refer to get_started.md for installation.