1. Installation

Installation

Aphrodite is primarily a python library, but also contains CUDA C/C++ code that needs to be compiled. We provide pre-compiled binaries in a wheel, so users don't have to deal with the compile times.

Requirements

1. Linux, Windows (WSL)
2. Python 3.8 - 3.11
3. NVIDIA (compute capability > 6.0) and AMD (MI200+)

Supported GPUs

Any NVIDIA GPU with a compute capability of 6.0 or higher. Refer to this page for a full list of CUDA GPUs:

https://developer.nvidia.com/cuda-gpus.

Or, you can manually find out your GPU's Compute Capability by opening a Python interpreter and running:

>>> import torch    # if you don't have `torch` installed, run `pip install torch` first
>>> print(torch.cuda.get_device_capability())

This should print something like this: (7, 5), which would indicate a CC of 7.5

If you do not meet the minimum CC, you will not be able to run Aphrodite. At the moment, compute capability of 7.5 or higher is required for AWQ quantization scheme; you can use GPTQ if your GPU does not support it

For AMD GPUs, only MI200 and above are supported at the moment. This means no RDNA GPUs (e.g. 7900 XTX). This is due to limitations from Flash Attention.

Pre-compiled binaries via PyPi

For the pip package, all you need is NVIDIA drivers.

$ pip install aphrodite-engine

Build from source

The embedded runtime sets up micromamba and downloads CUDA 11.8 along with all the required dependencies.

$ git clone https://github.com/PygmalionAI/aphrodite-engine.git && cd aphrodite-engine
$ ./update-runtime.sh  # this can take a while

You can then use the runtime.sh script to enter the Aphrodite Engine environment. You may also simply prefix all your Aphrodite-related commands with ./runtime.sh to execute them within the created env.

Alternatively, if you already have an environment with CUDA 11.8, you can simply run the following to build aphrodite-engine:

$ pip install -e .  # this can take a while