You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I tested three different datasets with different amounts in alpaca-lora and multi-lora-fine-tune.
Each dataset(the input data's sequence and size are also the same) trains two different lora models with two different optimizers, each optimizer has the same training hyperparams.
So the alpaca-lora needs to be trained twice to produce two different lora model, but multi-lora-fine-tune just need once to produce one lora model.
The experimental statistics on end-to-end train latency (without model and dataset load and save latency).
dataset1 use batchsize 7, 457 data from alpaca-lora, and max seq len is 1304
dataset2 use batchsize 16, 452 data from alpaca-lora, and max seq len is 512
dataset3 use batchsize 16, 5000 data from sql-create-context, and max seq len is 256
The experimental results are as follows:
Train two different lora total cost time(hours)
Train two different lora thuoughput(tokens/second)
The text was updated successfully, but these errors were encountered:
Test report about: alpaca_data_en_52k dataset on vicuna-7b-v1.1 (GPU: A100)
Method1: Using the same configuration file and data, fine-tune two datasets simultaneously.
Method2: Using the same configuration file and data, fine-tune only one dataset.
Method1: time cost: 12h45min, gpu memory cost: 23.69GB
Method2: time cost: 6h10min, gpu memory cost: 17.54GB
Test report about: alpaca_data_en_52k dataset on vicuna-7b-v1.1 (GPU: A100)
Method1: Using the same configuration file and data, fine-tune two datasets simultaneously. Method2: Using the same configuration file and data, fine-tune only one dataset.
Method1: time cost: 12h45min, gpu memory cost: 23.69GB Method2: time cost: 6h10min, gpu memory cost: 17.54GB
Does alpaca-lora and multi-lora-fine-tune set the group_by_length = true? Due to the random group of the dataset, the training time may be different, even in alpaca-lora, the training time varies greatly.
I tested three different datasets with different amounts in alpaca-lora and multi-lora-fine-tune.
Each dataset(the input data's sequence and size are also the same) trains two different lora models with two different optimizers, each optimizer has the same training hyperparams.
So the alpaca-lora needs to be trained twice to produce two different lora model, but multi-lora-fine-tune just need once to produce one lora model.
The experimental statistics on end-to-end train latency (without model and dataset load and save latency).
The experimental results are as follows:
The text was updated successfully, but these errors were encountered: