Diffusion Domain Teacher: Diffusion Guided Domain Adaptive Object Detector

This repository is the code implementation of the paper Diffusion Domain Teacher: Diffusion Guided Domain Adaptive Object Detector , which is based on the MMDetection project.

Installation

Requirements

• Linux system, Windows is not tested
• Python 3.8+, recommended 3.11
• PyTorch 2.0 or higher, recommended 2.0.0
• CUDA 11.7 or higher, recommended 11.8
• MMCV 2.0 or higher, recommended 2.0.0
• MMDetection 3.0 or higher, recommended 3.3.0
• diffusers 0.20.0 or higher, recommended 0.20.2

Environment Installation

It is recommended to use conda for installation. The following commands will create a virtual environment named DDT and install PyTorch and MMCV. In the following installation steps, the default installed CUDA version is 11.8. If your CUDA version is not 11.8, please modify it according to the actual situation. Note: If you are experienced with PyTorch and have already installed it, you can skip to the next section. Otherwise, you can follow the steps below.

Step 1: Create a virtual environment named DDT and activate it.

conda create -n DDT python=3.11 -y
conda activate DDT

Step 2: Install PyTorch2.x.

Linux/Windows:

conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia

Step 3: Install MMDetection-3.x.

pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0"
mim install mmdet=3.3.0

Step 4: Prepare for Stable-diffusion-1.5 with diffusers

git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/runwayml/stable-diffusion-v1-5

And you should move the Stable-diffusion-1.5 to the same dir as our DDT. Then:

pip install diffusers==0.20.2

Dataset Preparation

Cross-domian detection datasets

We provide the cross-domain detection dataset used in the paper. To facilitate the fast implementation of code, we provide Coco-style json files for all datasets google drive.

• Image and annotation download link: Cityscapes.
• Image and annotation download link: BDD 100k.
• Image and annotation download link: SIM10k.
• Image and annotation download link: VOC 07+12.
• Image and annotation download link: Clipart, Comic, Watercolor.

Code of DDT

Important code directories：

• DA：The root directory of our config file for DDT.
• DA/_base_/da_setting：Training and lr config of DDT.
• DA/_base_/datasets：Six cross-domain datasets config of DDT.
• DA/Ours：Detector config on six cross-domian datastes of DDT.
• mmdet/models/backbones/dift_encoder.py：Code and setting of diffusion backbone.
• mmdet/models/detectors/Z_dift_semi_base.py：Code of diffusion teacher for self-training.
• mmdet/models/detectors/Z_domain_adaptation_detector.py：Main code of DDT.
• mmdet/engine/hooks/adaptive_teacher_hook.py：Training and testing hooks of DDT.

Model Training

Diffusion detector training

The models are trained for 20,000 steps on two 3090 GPUs, with a batch size of 16. If your settings are different from ours, please modify the training steps and default learning rate settings in training config. Or You can use the trained models that we provide google drive.

Multi-gpu Training

sh ./tools/dist_train.sh ${CHECKPOINT_FILE} ${GPU_NUM}  # CHECKPOINT_FILE is the configuration file you want to use, GPU_NUM is the number of GPUs used

For example:

sh ./tools/dist_train.sh DA/Ours/city_to_bdd100k/diffusion_faster-rcnn_fpn_city_to_bdd100k_source.py  2  

Diffusion Domain Teacher

The models are trained for 20,000 steps on two 3090 GPUs, with a batch size of 16. We train exclusively on the source domain for the first 12000 steps and then perform joint training on both the source and target domain for the remaining 8000 steps.

If your settings are different from ours, please modify the training steps and default learning rate settings in training config. You should first modify the config and weights paths here, ensuring that the differential detector being used matches your location.

For example:

sh ./tools/dist_train.sh DA/Ours/city_to_bdd100k/DDT_r101_fpn_city_to_bdd100k.py 2 

Model Testing

Multi-gpu Testing：

Note: Please change the code here detector.dift_model.config and detector.dift_model.pretrained_model as None before test.

sh ./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM}  # CONFIG_FILE is the configuration file you want to use, CHECKPOINT_FILE is the checkpoint file you want to use, GPU_NUM is the number of GPUs used

Trained models

• Cityscapes to BDD100k: google drive
• Sim10k to BDD 100k: google drive
• Sim10k to Cityscapes: google drive
• VOC to Clipart: google drive
• VOC to Comic: google drive
• VOC to Watercolor: google drive