USAGE:

python infer.py --config [CONFIG_FILE]

Requirements:

1. python 3.10
2. pip3 install torch torchvision
3. pip3 install transformers==4.48.2
4. pip3 install tiktoken
5. pip3 install accelerate
6. pip3 install flash-attn --no-build-isolation
7. pip3 install vllm

Gemma

need to setup HF Token

export HF_TOKEN=[YOUR TOKEN]

or do:

hf auth login