Exploring Human-in-the-Loop Test-Time Adaptation by Synergizing Active Learning and Model Selection

This repository provides the official PyTorch implementation of our TMLR paper:

[Exploring Human-in-the-Loop Test-Time Adaptation by Synergizing Active Learning and Model Selection]
Authors: Yushu Li*, Yongyi Su*, Xulei Yang, Kui Jia, Xun Xu†

🔍 Overview

The overall framework of HILTTA.

Human-in-the-Loop Test-Time Adaptation (HILTTA) is a plug-and-play method that enhances Test-Time Adaptation (TTA) by integrating active learning and model selection. By employing a limited label budget, HILTTA achieves improved robustness to hyperparameter variations and enhanced performance in continually changing scenarios. It incorporates K-Margin for active sample selection and a model selection mechanism.

This repository provides the implementation of HILTTA, demonstrating its integration with six state-of-the-art TTA methods across four datasets: CIFAR10-C, CIFAR100-C, ImageNet-C, and ImageNet-D."

📐 Prerequisites

Hardware

This implementation supports single-GPU configurations. We recommend an NVIDIA RTX 3090 with 24GB of memory for optimal performance, as used in our experiments.

Environment

The code is tested on Python 3.8. Please install the required packages using the following command:

pip install -r requirements.txt

Datasets

We use the following datasets for evaluation:

• cifar10_c CIFAR10-C
• cifar100_c CIFAR100-C
• imagenet_c ImageNet-C
• imagenet_d ImageNet-D

Please refer to the test-time-adaptation repository for data preparation details. And download datasets to the data folder, listed as below:

data/
├── CIFAR-10-C/
├── CIFAR-100-C/
├── ImageNet-C/
├── imagenet2012/
│   └── val/
└── imagenet-d/

Models

• For adapting to ImageNet variations, all pre-trained models available in Torchvision or timm can be used.
• For the corruption benchmarks, pre-trained models from RobustBench can be used.

🚀 Run HILTTA

This section details how to execute HILTTA using the provided run.sh script.

Usage:

bash run.sh -m [method] -d [dataset]

options:
  -m            Choose from the following methods: 
                    ["tent" "pl" " "shot" "sar" "eata" "rmt"]
  -d            Choose from the following datasets: 
                    ["cifar10-c" "cifar100-c" "imagenet-c" "imagenet-d"]

For example, to run HILTTA with the TENT method on the CIFAR10-C dataset, execute the following command:

bash run.sh -m tent -d cifar10-c

📜 Citation

If you find our code useful or our work relevant, please consider citing:

@article{li2024exploring,
    title={Exploring Human-in-the-Loop Test-Time Adaptation by Synergizing Active Learning and Model Selection},
    author={Yushu Li and Yongyi Su and Xulei Yang and Kui Jia and Xun Xu},
    journal={Transactions on Machine Learning Research},
    issn={2835-8856},
    year={2024},
    url={https://openreview.net/forum?id=P09rAv8UH7}
}

🙏 Acknowledgements

We extend our sincere gratitude to the authors of the following repositories for their valuable open-source implementations and data preparation guidance:

• RMT
• TENT
• SHOT
• SAR
• EATA