Skip to content

ruz048/AutoLoRA

Repository files navigation

AutoLoRA

This folder contains the implementation of AutoLoRA ("AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning") in RoBERTa.

Paper Link: https://arxiv.org/pdf/2403.09113.pdf

Steps to reproduce our results

Install the pre-requisites

pip install -r requirements.txt

(Note that the source code of transformer library and Betty library AutoLoRA uses is different from that in the main branch)

Start the experiments

Search for the optimal rank of RoBERTa-base model on CoLA dataset

sh glue_search.sh

After obtaining optimal rank list, define the optimal ranks r_list in finetune/peft/utils/globals.py For example:

r_list=[3, 2, 4, 4, 5, 3, 3, 4, 5, 3, 5, 4, 3, 3, 5, 4, 2, 5, 4, 4, 3, 4, 3, 3] 

Finetuning the RoBERTa-base model with optimal LoRA rank:

python lora_glue.py

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages