Skip to content

liketheflower/UODDM

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

37 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

UODDM

Related to our paper Unified Object Detector for Different Modalities based on Vision Transformers. Our unified model can process RGB images, pseudo images converted from point clouds or inter-modality mixing of RGB image and pseudo images converted from point clouds.
Comparison of other systems can be seen here This repo contains the supported code and configuration files to reproduce object detection results of [simCrossTrans]. It is modified based on Swin Transformer for object detection. Original Readme

MODELs

Pretained based on COCO dataset

Those are the base model used for the UODDM work. You can download them from the official SWIN transformer repo, you can also download a backup from the link(Google Drive) provided here:

Finetune dataset model checkpoint
COCO Swin-T swin-t-model
COCO Swin-S swin-s-model

UODDM finetune on SUN RGBD dataset

The UODDM work was finetuning the above model based on SUN RGBD dataset. It has two models based on different modalities:

  • INPUT A: RGB.
  • INPUT B: RGB and DHS and RGB DHS mixed based on chessboard mixture.

We also had a input only as DHS model can be found in the simCrossTrans work. Here the performance based on mAP50 for SUNRGBD10, which includes a 10 common categories. Details please check the UODDM paper.

Finetune dataset input model checkpoint configure file performance on RGB validation performance on DHS validation performance when both RGB and DHS are available log
SUN RGBD INPUT A swin-t basedRGB cfg 53.9 N/A N/A log
SUN RGBD INPUT B swin-t basedRGBandDHSandRGBDHSmixed cfg 54.2 55.8 58.1 test_on_RGB test on RGB DHS mixed

Usage

Finetune based on COCO for SUNRGBD

The sun rgbd dataset training and test can be found in the sunrgbd folder, if you want to train the sunrgbd dataset based on pretrained model on COCO, please do the following:

cd sunrgbd
./shell_script/uoddm/train_swin_transform.sh

You need download a pretrained model from the COCO dataset and you can find the models in the MODEL session. If you want to train a RGB image, please use:# train RGB with pretrained weights from coco for 100 epochs"

About SUN RGBD categories

The SUN RGBD dataset also has 80 categories to align with COCO dataset. The SUNRGBD is direclty overwritten the COCO dataset's class, see this line If you want to directly use the pretrained model from SUN RGBD dataset, you need use the following customized mmdetection (updating the categories name to SUN RGBD and add some inference code).

https://github.com/liketheflower/mmdetection_beta

Inference

Run the following shell script:

./sunrgbd/shell_script/uoddm/inference/inference.sh

Installation

Please refer to get_started.md for installation.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages