A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis

We provide the first fine-grained and up-to-date study on the transferability of various brand-new pre-training techniques for medical imaging tasks. In particular, we conduct a systematic study on the transferability of (1) models pre-trained on iNat2021, the most recent large-scale fine-grained dataset, (2) 14 top self-supervised ImageNet models, and (3) domain-adapted models that are continuously pre-trained on natural and medical datasets.

In this repository we release multiple models from our paper as well as the code to fine-tuning the released models in desired target tasks. We hope that in the light of our study, the medical imaging community will benefit from the use of more powerful pre-trained models than conventional supervised models pre-trained on the ImageNet dataset.

Publication

A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis

Mohammad Reza Hosseinzadeh Taher¹, Fatemeh Haghighi¹, Ruibin Feng², Michael B. Gotway³, Jianming Liang^1
1 Arizona State University, ² Stanford University, ³ Mayo Clinic
International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021); Domain Adaptation and Representation Transfer (DART)

Paper | Code | Poster | Slides | Presentation (YouTube, YouKu)

Major results from our work

1. Pre-trained models on fine-grained data are better suited for segmentation tasks, while pre-trained models on coarse-grained data prevail on classification tasks.

2. Self-supervised ImageNet models outperform supervised ImageNet models in medical imaging tasks.

3. Domain-adaptive pre-training bridges the gap between the natural and medical imaging domains.

Installation

Clone the repository and install dependencies using the following command:

$ git clone https://github.com/MR-HosseinzadehTaher/BenchmarkTransferLearning.git
$ cd BenchmarkTransferLearning/
$ pip install -r requirements.txt

Model Zoo

In this paper, we evaluate 21 pre-trained ResNet50 models, including: 1 supervised ImageNet model, 14 self-supervised ImageNet models, 2 supervised iNat models, 2 in-domain pre-trained models, and 2 domain-adapted pre-trained models. We prepare self-supervised ImageNet models by referring to the released code by Linus Ericsson et al. To download and prepare all models in the same format, run:

python download_and_prepare_models.py

Note 1: iNat2021 models must be downloaded manually (using the links in the table below).

Note 2: For SimCLR-v1 and SimCLR-v2, the TensorFlow checkpoints need to be downloaded manually (using the links in the table below) and converted into PyTorch format (using https://github.com/tonylins/simclr-converter and https://github.com/Separius/SimCLRv2-Pytorch, respectively).

Note 3: For converting BYOL, you need to install some extra packages by running:

pip install jax jaxlib dill git+https://github.com/deepmind/dm-haiku

You can download the pretrained models used/developed in our paper as follows:

Category	Backbone	Platform	Training Dataset	Training Objective	model
In-domain models	ResNet50	Pytorch	ChestX-ray14	Supervised	download
	ResNet50	Pytorch	ChexPert	Supervised	download
Domain-adapted models	ResNet50	Pytorch	ImageNet → ChestX-ray14	Supervised → Supervised	download
	ResNet50	Pytorch	ImageNet → ChexPert	Supervised → Supervised	download
iNat2021 models	ResNet50	Pytorch	iNat2021	Supervised	download
	ResNet50	Pytorch	iNat2021 mini	Supervised
Self-supervised models	ResNet50	Pytorch	ImageNet	Barlow Twins	download
	ResNet50	Pytorch	ImageNet	SwAV	download
	ResNet50	Pytorch	ImageNet	DeepCluster-v2	download
	ResNet50	Pytorch	ImageNet	Sela-v2	download
	ResNet50	Pytorch	ImageNet	InsDis	download
	ResNet50	Pytorch	ImageNet	MoCo-v1	download
	ResNet50	Pytorch	ImageNet	MoCo-v2	download
	ResNet50	Pytorch	ImageNet	PCL-v1	download
	ResNet50	Pytorch	ImageNet	PCL-v2	download
	ResNet50	Pytorch	ImageNet	SimCLR-v1	download
	ResNet50	Pytorch	ImageNet	SimCLR-v2	download
	ResNet50	Pytorch	ImageNet	InfoMin	download
	ResNet50	Pytorch	ImageNet	PIRL	download
	ResNet50	Pytorch	ImageNet	BYOL	download

For the iNat21 and self-supervised ImageNet models, we used the official publicly available pre-trained models from the original papers, so, we give credit to the corresponding authors.

Datasets

Datasets can be downloaded from the following links.

ChestX-ray14

ChexPert

DRIVE (pre-processed dataset can be downloaded from here)

Montgomery

Shenzhen

RSNA PE detection

SIIM-ACR Pneumothorax Segmentation

For the classificatin tasks, we have provided data splits in the dataset directory. For each of train, test, and validation sets, a text file including images and labels information is provided. For the segmentation tasks, you need to put the train, test, and validation sets into separate folders.

Fine-tuing of pre-trained models on target task

1. Download the desired pre-trained model from the above-mentioned links.
2. Download the desired dataset; you can simply add any other dataset that you wish.
3. For classification tasks, run the following command by the desired parameters. For example, to finetune barlowtwins on ChestX-ray14, run:

python main_classification.py --data_set ChestXray14  \
--init barlowtwins \
--proxy_dir path/to/pre-trained-model \
--data_dir path/to/dataset \
--train_list dataset/Xray14_train_official.txt \
--val_list dataset/Xray14_val_official.txt \
--test_list dataset/Xray14_test_official.txt 

Or, to evaluate supervised ImageNet model on ChestX-ray14, run:

python main_classification.py --data_set ChestXray14  \
--init ImageNet \
--data_dir path/to/dataset \
--train_list dataset/Xray14_train_official.txt \
--val_list dataset/Xray14_val_official.txt \
--test_list dataset/Xray14_test_official.txt 

4. For segmentation tasks, run the following command by the desired parameters. For example, to finetune barlowtwins on DRIVE, run:

python main_segmentation.py --data_set DRIVE  \
--init barlowtwins \
--proxy_dir path/to/pre-trained-model \
--train_data_dir path/to/train/images \
--train_mask_dir path/to/train/masks \
--valid_data_dir path/to/validation/images \
--valid_mask_dir path/to/validation/masks \
--test_data_dir path/to/test/images \
--test_mask_dir path/to/test/masks

Or, to evaluate supervised ImageNet model on DRIVE, run:

python main_segmentation.py --data_set DRIVE  \
--init ImageNet \
--train_data_dir path/to/train/images \
--train_mask_dir path/to/train/masks \
--valid_data_dir path/to/validation/images \
--valid_mask_dir path/to/validation/masks \
--test_data_dir path/to/test/images \
--test_mask_dir path/to/test/masks

Citation

If you use this code or use our pre-trained weights for your research, please cite our paper:

@InProceedings{Hosseinzadeh2021Systematic,
 author="Hosseinzadeh Taher, Mohammad Reza and Haghighi, Fatemeh and Feng, Ruibin and Gotway, Michael B. and Liang, Jianming",
 title="A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis",
 booktitle="Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health",
 year="2021",
 publisher="Springer International Publishing",
 address="Cham",
 pages="3--13",
 isbn="978-3-030-87722-4"
}

Acknowledgement

This research has been supported partially by ASU and Mayo Clinic through a Seed Grant and an Innovation Grant, and partially by the NIH under Award Number R01HL128785. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. This work has utilized the GPUs provided partially by the ASU Research Computing and partially by the Extreme Science and Engineering Discovery Environment (XSEDE) funded by the National Science Foundation (NSF) under grant number ACI-1548562. We thank Nahid Islam for evaluating the self-supervised methods on the PE detection target task. The content of this paper is covered by patents pending. We build U-Net architecture for segmentation tasks by referring to the released code at segmentation_models.pytorch.

License

Released under the ASU GitHub Project License.