SubTuning is a parameter-efficient method for fine-tuning pretrained neural networks. It selectively trains specific layers while keeping others at their pre-trained state, optimizing performance based on the task and data distribution. Compared to conventional fine-tuning, SubTuning excels in scenarios with scarce or corrupted data and often matches the performance of traditional fine-tuning with abundant data. It has been successfully applied across various tasks, architectures, and pre-training methods.
To run experiments, follow these steps:
- Clone the repository.
- Build and run the Docker container by running the following commands:
cd docker
docker build -t subtuningimage .
docker run --rm -it subtuningimage
-
insert your WandB entity, API key, and project name in the wandb.text file located in the project directory.
-
run experiments using the provided bash scripts. The hyperparameters in the bash scripts can be modified. For instance, by changing the value of model.layers_to_finetune, you can choose the layers to fine-tune. For example, to create a finetune profile for ResNet50, run the following command:
bash scripts/greedy_subtuning_vit_cifar100.sh
To create the VTAB-1k data, run the get_pytorch_versions_of_vtab.py script.
If you have any issues or questions, please create a git issue.
@misc{https://doi.org/10.48550/arxiv.2302.06354,
doi = {10.48550/ARXIV.2302.06354},
url = {https://arxiv.org/abs/2302.06354},
author = {Kaplun, Gal and Gurevich, Andrey and Swisa, Tal and David, Mazor and Shalev-Shwartz, Shai and Malach, Eran},
keywords = {Machine Learning (cs.LG), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {SubTuning: Efficient Finetuning for Multi-Task Learning},
publisher = {arXiv},
year = {2023},
copyright = {Creative Commons Attribution Non Commercial No Derivatives 4.0 International}
}