Unsupervised Domain Adaptation in Semantic Segmentation via Orthogonal and Clustered Embeddings

This is the official PyTorch implementation of our work: "Unsupervised Domain Adaptation in Semantic Segmentation via Orthogonal and Clustered Embeddings" accepted at WACV 2021.

In this paper we propose an effective Unsupervised Domain Adaptation (UDA) strategy, based on feature clustering, orthogonality and sparsity objectives to reach a regularized disposition of latent embeddings , while providing a semantically consistent domain alignment over the feature space. We evaluate our framework in the synthetic-to-real scenario, in particular with GTA5, SYNTHIA and Cityscapes datasets.

The webpage of the paper is: here

The 5 minutes presentation video is: here

Slides of the presentation are: here

Requirements

This repository uses the following libraries:

• python (3.6)
• pytorch (1.0.0)
• torchvision (0.2.1)
• torchsummary (1.5.1)
• tensorboardX (1.4)
• tqdm (4.26.0)
• imageio (2.1.2)
• numpy (1.14.6)
• pillow (5.3.0)

A guide to install Pytorch can be found here.

For an easier installation of the other libraries, we provide you the requirement file to be run

pip install -r requirements.txt

Setup

GTA5

Download the GTA5 dataset and place it inside the datasets/GTA5 folder. The dataset split is provided within the train.txt and val.txt files in that folder.

Cityscapes

Download the Cityscapes dataset and place it inside the datasets/Cityscapes folder. The dataset split is provided within the train.txt, val.txt and test.txt files in that folder.

SYNTHIA

Download the SYNTHIA-RAND-CITYSCAPES dataset and place it inside the datasets/SYNTHIA folder. The dataset split is provided within the train.txt and val.txt files in that folder.

Training

Source Pre-training (example)

GTA5 with ResNet101:

Download the ResNet101's weights pretrained on ImageNet from here and run

python tools/train_source.py --backbone "resnet101" --dataset "gta5" --num_classes 19 --checkpoint_dir "./log/gta5-resnet_pretrain/" --iter_max 200000 --iter_stop 80000 --lr 2.5e-4 --crop_size "1280,720"

Alternatively, you can download the GTA5 pretrained weights from here.

Unsupervised Domain Adaptation

• GTA5-to-Cityscapes with ResNet101:

  python tools/train_UDA.py --source_dataset "gta5" --num_classes 19 --backbone "resnet101" --round_num 5 --lr 2.5e-4 --lambda_cluster 2e-2 --lambda_ortho 2. --lambda_sparse 5e-1 --lambda_entropy 1e-1 --target_crop_size "1024,512" --crop_size "1280,720" --checkpoint_dir "./log/gta2city-resnet_UDA/" --pretrained_ckpt_file "./log/gta5-resnet_pretrain/gta5final.pth"

  A checkpoint of the adapted model can be found here.

• GTA5-to-Cityscapes with VGG-16:

  python tools/train_UDA.py --source_dataset "gta5" --num_classes 19 --backbone "vgg16" --round_num 5 --lr 2.5e-4 --lambda_cluster 1e-1 --lambda_ortho 2e-2 --lambda_sparse 2e-2 --lambda_entropy 2e-2 --target_crop_size "1024,512" --crop_size "1280,720" --checkpoint_dir "./log/gta2city-vgg_UDA/" --pretrained_ckpt_file "./log/gta5-vgg_pretrain/gta5final.pth"

  A checkpoint of the adapted model can be found here.

• SYNTHIA-to-Cityscapes with ResNet-101:

  python tools/train_UDA.py --source_dataset "synthia" --num_classes 16 --backbone "resnet101" --round_num 5 --lr 2.5e-4 --lambda_cluster 2e-2 --lambda_ortho 1e-1 --lambda_sparse 1 --lambda_entropy 5e-2 --IW_ratio 0. --target_crop_size "1024,512" --crop_size "1280,760" --checkpoint_dir "./log/sy2city-vgg_UDA/" --pretrained_ckpt_file "./log/synthia-vgg_pretrain/synthiafinal.pth"

  A checkpoint of the adapted model can be found here.

• SYNTHIA-to-Cityscapes with VGG16:

  python tools/train_UDA.py --source_dataset "synthia" --num_classes 16 --backbone "vgg16" --round_num 5 --lr 2.5e-4 --lambda_cluster 5e-2 --lambda_ortho 5e-2 --lambda_sparse 2e-2 --lambda_entropy 1e-2 --target_crop_size "1024,512" --crop_size "1280,760" --checkpoint_dir "./log/sy2city-vgg_UDA/" --pretrained_ckpt_file "./log/synthia-vgg_pretrain/synthiafinal.pth"

  A checkpoint of the adapted model can be found here.

Evaluation (example)

GTA5-to-Cityscapes with ResNet101:

python tools/evaluate.py --source_dataset "gta5" --num_classes 19 --backbone "resnet101" --split "test" --target_crop_size "1024,512" --checkpoint_dir "./log/eval/gta2city-resnet_UDA/" --pretrained_ckpt_file "./log/gta2city-resnet_UDA/gta52cityscapesfinal.pth"

Cite us

If you use this repository, please consider to cite

   @inProceedings{toldo2021unsupervised,
   author = {Toldo, Marco and Michieli, Umberto and Zanuttigh, Pietro},
   title  = {Unsupervised Domain Adaptation in Semantic Segmentation via Orthogonal and Clustered Embeddings},
   booktitle = {Winter Conference on Applications of Computer Vision (WACV)},
   year      = {2021},
   month     = {January}
   }

Acknowledgments

The structure of this code is largely based on this repo.

License

Apache License 2.0