Model-Agnostic Adapters [ICCV 2023 Oral]

Official implementation of the paper "MAP: Towards Balanced Generalization of IID and OOD through Model-Agnostic Adapters".

---

Highlights

Abstract: Deep learning has achieved tremendous success in recent years, but most of these successes are built on an independent and identically distributed (IID) assumption. This somewhat hinders the application of deep learning to the more challenging out-of-distribution (OOD) scenarios. Although many OOD methods have been proposed to address this problem and have obtained good performance on testing data that is of major shifts with training distributions, interestingly, we experimentally find that these methods achieve excellent OOD performance by making a great sacrifice of the IID performance. We call this finding the IID-OOD dilemma. Clearly, in real-world applications, distribution shifts between training and testing data are often uncertain, where shifts could be minor, and even close to the IID scenario, and thus it is truly important to design a deep model with the balanced generalization ability between IID and OOD. To this end, in this paper, we investigate an intriguing problem of balancing IID and OOD generalizations and propose a novel Model Agnostic adaPters (MAP) method, which is more reliable and effective for distribution-shift-agnostic real-world data. Our key technical contribution is to use auxiliary adapter layers to incorporate the inductive bias of IID into OOD methods. To achieve this goal, we apply a bilevel optimization to explicitly model and optimize the coupling relationship between the OOD model and auxiliary adapter layers. We also theoretically give a first-order approximation to save computational time. Experimental results on six datasets successfully demonstrate that MAP can greatly improve the performance of IID while achieving good OOD performance.

Supported Methods

The currently available algorithms are:

• Empirical Risk Minimization (ERM, Vapnik, 1998).
• Invariant Risk Minimization (IRM, Arjovsky et al., 2019).
• Variance Risk Extrapolation (VREx, Krueger et al., 2020), contributed by @zdhNarsil.
• Adaptive Risk Minimization (ARM, Zhang et al., 2020), contributed by @zhangmarvin.
• Group Distributionally Robust Optimization (GroupDRO, Sagawa et al., 2020).
• Meta Learning Domain Generalization (MLDG, Li et al., 2017).
• Maximum Mean Discrepancy (MMD, Li et al., 2018).
• Out-of-Distribution Generalization with Maximal Invariant Predictor (IGA, Koyama et al., 2020).
• Smoothed-AND mask (SAND-mask, Shahtalebi et al., 2021).
• Gradient Matching for Domain Generalization (Fish, Shi et al., 2021).
• Conditional Domain Adversarial Neural Network (CDANN, Li et al., 2018).
• Learning Representations that Support Robust Transfer of Predictors (TRM, Xu et al., 2021).
• Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization (IB-ERM , Ahuja et al., 2021).
• Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization (IB-IRM, Ahuja et al., 2021).
• Optimal Representations for Covariate Shift (CAD & CondCAD, Ruan et al., 2022), contributed by @ryoungj.
• Causality based invariant representation learning algorithm using the CORAL distance from (https://arxiv.org/abs/2206.11646) [CausIRL_CORAL].

Results

MAP in comparison with 16 existing methods on 6 OOD benchmarks

Environment requirements

• Python 3.6 or above
• The packages listed in requirements.txt. You can install them via pip install -r requirements.txt. Package torch_scatter may require a manual installation
• Submodules are added to the path:

export PYTHONPATH="$PYTHONPATH:$(pwd)/external/DomainBed/"
export PYTHONPATH="$PYTHONPATH:$(pwd)/external/wilds/"

Data preparation

Please follow this instruction for three toy datasets and three real datasets. Dataset list:

• ColoredMNIST
• ColoredCOCO
• COCOPlaces
• NICO
• CelebA
• WILDSCamelyon

Training and Evaluation

Run the following command to train and evaluate MAP or please refer to README.md for more details.

Train a model

python -m domainbed.train\
        --data_dir=./datasets/MNIST/\
        --algorithm=IRM_MAP\
        --dataset=ColoredMNIST\
        --test_env=2

Launch a sweep:

python -m domainbed.sweep launch\
        --data_dir=./datasets/MNIST/\
        --algorithm=IRM_MAP\
        --output_dir=./results/\
        --command_launcher=local

To view the results of your sweep:

python -m domainbed.scripts.collect_results\
        --input_dir=./results/\

Citation

If you find the code useful or find our paper relevant to your research, please consider citing:

@inproceedings{zhang2023map,
  title={MAP: Towards Balanced Generalization of IID and OOD through Model-Agnostic Adapters},
  author={Zhang, Min and Yuan, Junkun and He, Yue and Li, Wenbin and Chen, Zhengyu and Kuang, Kun},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision, {ICCV}},
  pages={11921--11931},
  year={2023}
}

Contact

If you have any questions, please create an issue on this repository or contact at zhangmin.milab@zju.edu.cn.

Acknowledgement

Our code is based on Domainbed and OoD-Bench repository. We thank the authors for releasing their code. If you use our model and code, please consider citing these works as well.