Skip to content

The aim of this repository is to utilize LLaMA to reproduce and enhance the Stanford Alpaca

Notifications You must be signed in to change notification settings

dropreg/efficient_alpaca

Repository files navigation

Efficient-Alpaca

Efficient Alpaca

English | 中文

The aim of Efficient Alpaca is to utilize LLaMA to build and enhance the LLM-based chatbots, including but not limited to reducing resource consumption (GPU memory or training time), improving inference speed, and more facilitating researchers' use (especially for fairseq users). This project will be constantly updated and maintained. Please feel free to use it!

**************************** Updates ****************************

  • 4/5 We support Fine-tuning using FSDP to reduce GPU memory with extra RAM memory !
  • 3/17 We support model parallel to reduce GPU memory using Megatron-LM !
  • 3/15 We support LoRA (Efficient-finetuning) to reproduce Stanford Alpaca !

Supported Inference Devices

we can choose following device to support inference, even 12G 1080.

Method Support Device GPU Inference Speed
Original all 1 24G 3090 14G
Megatron megatron_lora 2 12G 1080 8G

Supported Training Methods and Devices

We can choose the following available methods and combinations: for example, I have 2 24G 3090 and a lot of memory, at this time you can have two options: 1. Use Megatron-LM for Efficient-Finetuning (does not use a lot of memory) . 2. Use FSDP for Fine-tuning (it will use a lot of extra memory).

Method Type Support Data Para Model Para Device GPU Memory Limit Training Speed
LoRA Efficient Fine-tuning lora 1 40G A100 30G No 90 sec / 100 step
Megatron-LoRA Efficient Fine-tuning megatron_lora 2 24G 3090 21G No 190 sec / 100 step
FSDP Fine-tuning fsdp 1 40G A100 32G 128G + 1600 sec / 100 step
8 40G A100 32G No 400 sec / 100 step
2 24G 3090 13G 128G + 900 sec / 100 step
8 24G 3090 22G 128G + 800 sec / 100 step
Megatron Fine-tuning megatron 4 40G A100 25G No 130 sec / 100 step
8 24G 3090 14G No 130 sec / 100 step
Some Explanation about Support Table.

All evaluation used hyper-parameter --max-tokens 2048.

  • Data Para: Whether to support data parallel.
  • Model Para: Whether to support model parallel.
  • GPU: GPU Memory usage of each node during training.
  • Memory Limit: RAM Memory usage during training.
  • Training Speed: Only represents training speed rather than training time, because data parallel supports to accelerate training.

Web Interface

We support web interface using Gradio

bash  alpaca_lora/scripts/run_webapp.sh

Examples

Setup

Ensure the pytorch and cuda environment available, and install fllowing dependences:

pip install fairseq
pip install fairscale

We have to install sentencepiece from official repo to process data or hack for your specific task.

git clone https://github.com/google/sentencepiece.git 
cd sentencepiece
mkdir build
cd build
cmake ..
make -j $(nproc)
make install

Prepare Model and Data

Training Step:

Efficient-Finetuning

Fine-tuning

Some Case Sampled by Our Model:

  1. Question: what is the capital of china?

    Answer: Beijing is the capital of China.

  2. Question: please give a pieces of python code about socket connection.

    Answer:

    import socket
    
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    print("Constructing a socket connection")
    print(s)
    
    s.bind(('', 5000)) # Bind the socket to the '(' and 5000'
    
    print("Listening for connections")
    while True:
        connection, address = s.accept()
        print("Got connection from", address)
        print("Connected!")
        connection.close() # Close the connection.
    
    print("Exiting the program")
    
  3. Question: 请讲一个武侠故事.

    Answer: The legendary warrior Guan Yu Once upon a time, there was a legendary warrior named Guan Yu. He was a highly skilled fighter and commander and was known for his great bravery and loyalty. He was the loyal servant of Emperor Liu Bei, and fought in the Three Kingdoms civil war alongside him. He was brave enough to challenge the powerful enemy warriors, and eventually helped Liu Bei's army defeat the Wei Kingdom, the strongest of the three kingdoms. Guan Yu was a great warrior and a hero in Chinese history.

  4. Question: what is an alpaca? how is it different form a llama?

    Answer: Alpacas and llamas are both members of the Camelidae family. They are both herbivorous mammals that are domesticated for their coats, but they are distinctly different species. Llamas are generally larger than alpacas and have more pronounced spiral horns, while alpacas have smaller horns and more delicate coats. Alpacas also have smaller teeth and a narrower neck than llamas, and they are able to reproduced year-round. Llamas are also unable to reproduce during the winter months and have a more social behavior than alpacas.

  5. Question: write a e-mail to congratulate Lee and mention that you are exited about meeting all of them in person.

    Answer:

    Dear Lee,
    
    I want to congratulate you on your success and I'm looking forward to meeting you and the other team members in person. 
    It's exciting to see the progress you've made and I'm sure the hard work you have put in has paid off. 
    I'm very proud of you and I'm looking forward to meeting you in person and seeing all of your progress.
    
    Sincerely,
    [Your Name]
    
  6. Question: write a well-thought our abstract for a machine learning parper that proves that 42 is the optimal seed for training neural network.

    Answer: This paper investigates the effect of different seeds on the performance of neural networks. Through experiments, we demonstrate that the optimal seed is 42, which leads to the best performance on a wide range of tasks. We analyze the effects of different seeds on the model's accuracy, robustness, and other metrics. Our results show that 42 is the best seed for neural networks, and we discuss the implications of this finding.

Reference

Fairseq: a sequence modeling toolkit [github]

@inproceedings{ott2019fairseq,
  title = {fairseq: A Fast, Extensible Toolkit for Sequence Modeling},
  author = {Myle Ott and Sergey Edunov and Alexei Baevski and Angela Fan and Sam Gross and Nathan Ng and David Grangier and Michael Auli},
  booktitle = {Proceedings of NAACL-HLT 2019: Demonstrations},
  year = {2019},
}

FairScale: is a PyTorch extension library for high performance and large scale training. [github]

@Misc{FairScale2021,
  author =       {{FairScale authors}},
  title =        {FairScale:  A general purpose modular PyTorch library for high performance and large scale training},
  howpublished = {\url{https://github.com/facebookresearch/fairscale}},
  year =         {2021}
}

LLaMA: Open and Efficient Foundation Language Models [paper][github]

@article{touvron2023llama,
  title={LLaMA: Open and Efficient Foundation Language Models},
  author={Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth{\'e}e and Rozi{\`e}re, Baptiste and Goyal, Naman and Hambro, Eric and Azhar, Faisal and Rodriguez, Aurelien and Joulin, Armand and Grave, Edouard and Lample, Guillaume},
  journal={arXiv preprint arXiv:2302.13971},
  year={2023}
}

Stanford Alpaca: An Instruction-following LLaMA model [github]

@misc{alpaca,
  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {Stanford Alpaca: An Instruction-following LLaMA model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}

About

The aim of this repository is to utilize LLaMA to reproduce and enhance the Stanford Alpaca

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published