Skip to content

Inference Codes for LLaMA with Intel Extension for Pytorch (Intel Arc GPU)

License

Notifications You must be signed in to change notification settings

Aloereed/llama-ipex

Repository files navigation

LLaMA for Intel Arc GPU

Available for Intel Arc GPU.

Interact

torchrun --standalone --nproc_per_node 1 interact.py --ckpt_dir models/7B --tokenizer_path ./models/7B/tokenizer.model --max_batch_size 32 --max_seq_len 128

LLaMA

This repository is intended as a minimal, hackable and readable example to load LLaMA (arXiv) models and run inference. In order to download the checkpoints and tokenizer, fill this google form

Setup

In a conda env with pytorch / cuda available, run

pip install -r requirements.txt

Then in this repository

pip install -e .

Download

Once your request is approved, you will receive links to download the tokenizer and model files. Edit the download.sh script with the signed url provided in the email to download the model weights and tokenizer.

Inference

The provided example.py can be run on a single or multi-gpu node with torchrun and will output completions for two pre-defined prompts. Using TARGET_FOLDER as defined in download.sh:

torchrun --nproc_per_node MP example.py --ckpt_dir $TARGET_FOLDER/model_size --tokenizer_path $TARGET_FOLDER/tokenizer.model

Different models require different MP values:

Model MP
7B 1
13B 2
33B 4
65B 8

FAQ

Model Card

See MODEL_CARD.md

License

See the LICENSE file.

About

Inference Codes for LLaMA with Intel Extension for Pytorch (Intel Arc GPU)

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published