LLaMA

About • Installation • How To Use • Final results • Credits • License

About

This repository contains the end-to-end pipeline for training text generation model with PyTorch. The model was implemented is LLaMA.

See the task assignment here.

See wandb report with all experiments.

Installation

Follow these steps to install the project:

0. (Optional) Create and activate new environment using conda.

   # create env
   conda create -n llama python=3.11
   
   # activate env
   conda activate llama

1. Install all required packages.

   pip install -r requirements.txt
   

2. Download dataset and tokenizer.

   python prepare.py
   

Training

The model training contains of 2 stages. To reproduce results, train model using the following commands:

1. Train 101 epochs as pretrain on 256 len sequences

   python train.py writer.run_name="part1" dataloader.batch_size=3 trainer.seqs_from_sample=16
   model.rope_coef=1 model.max_seq_len=256 lr_scheduler.max_lr=5e-4

2. Train 10 epochs as fine-tune on 1024 len sequences

   python train.py writer.run_name="part2" dataloader.batch_size=3 trainer.seqs_from_sample=3
   model.rope_coef=0.25 model.max_seq_len=1024

It takes around 26 hours to train model from scratch on V100 GPU.

Final results

This results were obtained using argmax and language model:

            RTI
tinyMMLU    0.304

Credits

This repository is based on a PyTorch Project Template.

License