DeCUR: decoupling common & unique representations for multimodal self-supervision.

PyTorch implementation of DeCUR.

Pretrained models

Pretrain dataset	Full model	Backbone only
SSL4EO-S12	ResNet50-SAR/MS-ep100
GeoNRW	ResNet50-RGB/DEM-ep100
SUNRGBD	MiTB2-RGB/HHA-ep200	MiTB2-RGB, MiTB2-HHA
SUNRGBD	MiTB5-RGB/HHA-ep200	MiTB5-RGB, MiTB5-HHA

DeCUR Pretraining

Customize your multimodal dataset and your preferred model backbone in src/datasets/, src/models/ and src/pretrain_mm.py, and run

python pretrain_mm.py \
--dataset YOUR_DATASET \
--method PRETRAIN_METHOD \
--data1 /path/to/modality1 \
--data2 /path/to/modality2 \
--mode MODAL1 MODAL2 

Apart from DeCUR, we also support multimodal pretraining with SimCLR, CLIP, BarlowTwins and VICReg.

If you are using distributed training with slurm, we provide some example job submission scripts in src/scripts/pretrain.

Transfer Learning

Multilabel scene classification with ResNet50 on BigEarthNet-MM:

$ cd src/transfer_classification_BE
$ python linear_BE.py --backbone resnet50 --mode s1 s2 --pretrained /path/to/pretrained_weights

Semantic segmentation with FCN on GeoNRW:

$ cd src/transfer_segmentation_GEONRW
$ python GeoNRW_MM_FCN_RN50.py --backbone resnet50 --mode RGB DSM mask --pretrained /path/to/pretrained_weights

Semantic segmentation with CMX on SUNRGBD and NYUDv2:

$ cd src/transfer_segmentation_SUNRGBD
$ python convert_weights.py # convert pretrained weights to CMX format

Then please refer to https://github.com/huaaaliu/RGBX_Semantic_Segmentation.
Simply load the pretrained weights from our pretrained models. 

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Citation

@article{wang2023decur,
  title={DeCUR: decoupling common & unique representations for multimodal self-supervision},
  author={Yi Wang and Conrad M Albrecht and Nassim Ait Ali Braham and Chenying Liu and Zhitong Xiong and Xiao Xiang Zhu},
  journal={arXiv preprint arXiv:2309.05300},
  year={2023}
}