☎️🦙 callama

                  888 888                                 
                  888 888                                 
                  888 888                                 
 .d8888b  8888b.  888 888  8888b.  88888b.d88b.   8888b.  
d88P"        "88b 888 888     "88b 888 "888 "88b     "88b 
888      .d888888 888 888 .d888888 888  888  888 .d888888 
Y88b.    888  888 888 888 888  888 888  888  888 888  888 
 "Y8888P "Y888888 888 888 "Y888888 888  888  888 "Y888888 
                                                          
                                                          
                                                          

call llama.cpp directly, either locally or remotely.

Why make this?

I wanted to be able to call llama.cpp from Python, but I didn't want to use the llama-cpp-python wrapper because it automatically downloads and builds llama.cpp, which I didn't want to do. I like the simplicity of llama.cpp and its unix philosophy of doing one thing well, so I prefer to build llama.cpp myself and then call it from Python.

• One advantage of this is that I can use the latest version of llama.cpp, instead of the version that llama-cpp-python downloads.

I also wanted to be able to call llama.cpp remotely. It's easy as long as the llama.cpp server is already running on the remote. But I wanted to be able to spawn several llama.cpp processes on the remote, send queries to them, and then kill them when I'm done. The existing solutions such as llama-cpp-python and LMQL can only execute the llama.cpp server locally.

callama's level of abstraction

...
↓
hf:transformers
↓
llama.cpp → outdated OpenAI Drop-in Replacement API
↓
llama-cpp-python / [[[ callama ]]]
↓
oobabooga / ollama / ...
↓
		↓
	     litellm / web ui / ...
		↓
	     OpenAI Drop-in Replacement API

Why not use OpenAI's (drop-in replacement) API?

Because there are differences between llama.cpp and OpenAI's API. For example, OpenAI doesn't support the --grammar option but llama.cpp does. Using the llama.cpp API reminds the user that they can't simply change the model name (as an OpenAI drop-in replacement API would allow) and expect everything to work the same.

That being said, I use Pydantic to validate the flags sent to llama.cpp to create a server and to send queries to it. This means your IDE (e.g., VSCode) will display the available flags and their types when you use this library.

Installation

callama is a Python library, so as with any Python library, you can install it with pip. It is strongly recommended that you install it in a virtual environment (e.g., with venv or conda) to avoid conflicts with other libraries. I suggest conda. Here's how to setup a conda environment and install callama in it:

# create a conda environment and activate it
conda create --name callama python=3.11 -y && conda activate callama
# then install callama
pip uninstall callama pydantic -y && pip install git+https://github.com/ibehnam/callama.git --no-cache-dir

As a rule of thumb, never install software that you don't know how to uninstall later! To uninstall callama:

pip uninstall -y callama
# You can also delete the conda environment
conda deactivate
conda remove --name callama --all -y

How to Use

After installing callama, you have two options:

1. Run llama.cpp manually and send queries to it using callama.
2. (recommended) Spawn a llama.cpp server using callama and send queries to it.

Option 1: Run llama.cpp manually and send queries to it using callama

Option 2: Use callama server to spawn llama.cpp processes and send queries to them

After installing callama on the machine that you want to run llama.cpp on, you can start callama's API server like this:

callama server --server-path <full-path-to-llama.cpp-server> --models-path <fulll-path-to-llama.cpp-models>

For example, if you have installed llama.cpp in /path/to/downloads/llama.cpp and your GGUF model is in /path/to/downloads/llama.cpp/models/, you can start the server like this:

callama server --server-path /path/to/downloads/llama.cpp/server --models-path /path/to/downloads/llama.cpp/models/

• callama's API server can be run on any machine that has callama and llama.cpp installed on it. It doesn't have to be the same machine that you're running your Python code on.
  • If you run the server on a remote machine, make sure that the port that you're using is open on the remote machine's firewall. It's easier to use tailscale or ngrok to expose the port on the remote machine to the internet or to your tailnet.

Why call it "callama"?

Because it's a call to llama.cpp 😄