The test report compare to alpaca-lora.

[yezhengmao1]

I tested three different datasets with different amounts in alpaca-lora and multi-lora-fine-tune.
Each dataset(the input data's sequence and size are also the same) trains two different lora models with two different optimizers, each optimizer has the same training hyperparams.
So the alpaca-lora needs to be trained twice to produce two different lora model, but multi-lora-fine-tune just need once to produce one lora model.
The experimental statistics on end-to-end train latency (without model and dataset load and save latency).

• dataset1 use batchsize 7, 457 data from alpaca-lora, and max seq len is 1304
• dataset2 use batchsize 16, 452 data from alpaca-lora, and max seq len is 512
• dataset3 use batchsize 16, 5000 data from sql-create-context, and max seq len is 256
  The experimental results are as follows:

1. Train two different lora total cost time(hours)
   
2. Train two different lora thuoughput(tokens/second)
   

[merlintang]

how about the gpu memory usage comparison?

[merlintang]

how about the gpu utils information?

[LianxinGao]

Test report about: alpaca_data_en_52k dataset on vicuna-7b-v1.1 (GPU: A100)

Method1: Using the same configuration file and data, fine-tune two datasets simultaneously.
Method2: Using the same configuration file and data, fine-tune only one dataset.

Method1: time cost: 12h45min, gpu memory cost: 23.69GB
Method2: time cost: 6h10min, gpu memory cost: 17.54GB

[yezhengmao1]

Test report about: alpaca_data_en_52k dataset on vicuna-7b-v1.1 (GPU: A100)

Method1: Using the same configuration file and data, fine-tune two datasets simultaneously. Method2: Using the same configuration file and data, fine-tune only one dataset.

Method1: time cost: 12h45min, gpu memory cost: 23.69GB Method2: time cost: 6h10min, gpu memory cost: 17.54GB

Does alpaca-lora and multi-lora-fine-tune set the group_by_length = true? Due to the random group of the dataset, the training time may be different, even in alpaca-lora, the training time varies greatly.