Proper comparison between adapter-tuning, lora-tuning, prompt-tuning, and prefix-tuning?

[jzhang38]

Parameter-efficient methods have been studied extensively in (small or large) language models since BERT. Given those abundant prior works, why is there no controlled experiment comparing those different methods in the paper?

[gaopengpjlab]

Your suggestion is quite nice. Actually, we are going to carry out extensive studies of zero-init attention on adaptation tasks spanning different areas, such as, BERT, GPT, Stable Diffusion, CLIP and LLM. Due to the large-volume of experiments, this process may take a long time. Stay tuned.

[ChrisLiu6]

Hi, thank you for your interest in our work!

We have evaluated zero-init attention on CV and NLP benchmarks, here are the results:

• For image classification, we follow Visual Prompt Tuning to finetune ViT-B/16 pretrained on ImageNet21k on the vtab-1k benchmark:

• For Natural language understanding, we follow P-tuning V2 and evaluate our method on multiple tasks:

We will incorporate these results in the paper after revision, and related codes will also be released. A more holistic evaluation involving more tasks and areas is currently in progress and will come soon. Thank you again for your suggestion!