🦙 llama-api-server

A portable, self-provisioning llama.cpp API server toolkit.

Auto-detects your CPU, fetches the right binaries + the fastest GGUF model for your hardware, tunes thread count, and runs an OpenAI-compatible /v1/chat/completions API — with a systemd service that auto-starts on boot and self-heals on crash.

One command to bootstrap on any Linux or macOS box (x86_64 / arm64). No Docker. No GPU required.

Quick start · Benchmarks · API · Config · Contributing

Why this exists

Running local LLMs is usually a mess of manual steps: pick a build for your CPU, download a model that might not fit your RAM, guess the thread count, and babysit a foreground process. This repo turns that into:

git clone https://github.com/axe01010/llama-api-server.git ~/llama
cd ~/llama && ./setup.sh          # detect CPU → fetch binaries → wire everything
./server.sh start                 # API live at http://127.0.0.1:8080

…and you get a tuned, monitored, auto-restarting API endpoint.

Features

Feature	Detail
Portable	Runs on Linux & macOS, x86_64 & arm64. `setup.sh` detects the platform and downloads the matching official llama.cpp release.
CPU-optimized	Detects AVX-512 / AVX2 / SSE / NEON and picks the right binary variant. Auto-tunes thread count to physical cores (SMT-aware).
Model hub	`fetch-model.sh search "1B GGUF"` queries Hugging Face by downloads; `get <repo> Q4_K_M` downloads + verifies GGUF magic; `set <file>` swaps the live model.
OpenAI-compatible	Drop-in `/v1/chat/completions`, `/v1/models`, `/health`, `/props`. Point any OpenAI-SDK client at it.
Self-healing	systemd user service: `Restart=on-failure`, auto-enable on login, graceful SIGINT shutdown.
Benchmarked	Built-in `llama-bench` sweep picks the fastest config. See docs/BENCHMARKS.md.
Tiny footprint	Repo is just scripts + config (~30 KB). Binaries, models, and source are provisioned per-machine and gitignored.

Quick start

1. Clone & bootstrap

git clone https://github.com/axe01010/llama-api-server.git ~/llama
cd ~/llama
./setup.sh

setup.sh will:

Detect your OS / architecture / CPU capabilities.
Verify existing binaries or download the latest official llama.cpp release for your platform.
Generate config.sh (model, port, threads, context size).
Wire launcher scripts (llama-cli, llama-server, …) and env.sh.
Install + enable a systemd user service (Linux only).

2. Fetch a model

# search Hugging Face for the most-downloaded small GGUF models
./fetch-model.sh search "1B GGUF"

# download a specific model (quant optional, defaults to Q4_K_M)
./fetch-model.sh get bartowski/Llama-3.2-1B-Instruct-GGUF Q4_K_M

# make it the live API model
./fetch-model.sh set models/Llama-3.2-1B-Instruct-Q4_K_M.gguf

3. Start the API

./server.sh start          # or:  systemctl --user start llama-server

4. Call it

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":128}'

Or with the OpenAI Python SDK:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed")
print(client.chat.completions.create(
    model="llama",
    messages=[{"role":"user","content":"Hello!"}]
).choices[0].message.content)

Web UI

A zero-dependency single-page chat interface ships in web/index.html — no build step, no npm, no framework. It talks to the API same-origin and supports streaming, markdown code blocks, adjustable temperature / max-tokens / system prompt, and a live connection indicator.

Serve it with any static file server pointed at the repo root:

# option A: python (built in)
python3 -m http.server 8000

# option B: llama-server itself serves static files — open http://127.0.0.1:8080/index.html

Then open http://localhost:8000/web/index.html. Point it at your API base URL in the Settings panel (defaults to same-origin /). The UI is dependency-free vanilla JS using the SSE stream, so it works offline and drops into any OpenAI-compatible backend.

Examples

Copy-paste-ready clients in examples/:

Language	File	Run
Python (SDK)	`examples/python/chat.py`	`pip install openai && python3 examples/python/chat.py --stream`
Python (stdlib)	`examples/python/chat.py --raw`	no install needed
Node.js (SDK)	`examples/node/chat.mjs`	`npm i openai && node examples/node/chat.mjs --stream`
Shell	`examples/shell/chat.sh`	`bash examples/shell/chat.sh --stream`

All examples honor LLAMA_BASE_URL and LLAMA_MODEL env vars.

Docker

Build and run in a container — models are bind-mounted, not baked in:

docker build -t llama-api-server .
docker run --rm -p 8080:8080 -v "$PWD/models:/models:ro" \
  -e MODEL=/models/qwen2.5-1.5b-instruct-q4_k_m.gguf llama-api-server

Or with Compose (reads MODEL / THREADS / CTX from env or .env):

MODEL=qwen2.5-1.5b-instruct-q4_k_m.gguf docker compose up -d

The multi-stage build keeps the image under ~80 MB. The HEALTHCHECK waits for the model to finish loading before reporting healthy.

Testing

# lint (syntax + shellcheck)
bash -n setup.sh fetch-model.sh server.sh

# end-to-end: boots a real server, hits every endpoint, validates responses
./tests/integration.sh

# pass an explicit model + port
./tests/integration.sh ./models/Llama-3.2-1B-Instruct-Q4_K_M.gguf 8090

The integration test auto-discovers a model from ./models or ~/llama/models; it skips (doesn't fail) if none is found. CI runs lint, a multi-OS setup.sh --check, the Hugging Face search probe, the full integration suite, and a Docker build.

How it works

Architecture

Data flow

flowchart LR
    U["Your app / curl"] -->|"POST /v1/chat/completions"| A["API server<br/>llama-server<br/>127.0.0.1:8080"]
    A -->|"read model"| M["models/*.gguf"]
    A -->|"CPU inference (AVX-512)"| B["bin/llama-server<br/>+ libggml-cpu-icelake.so"]
    C["config.sh"] -->|"LLAMA_MODEL / THREADS / CTX"| A
    S["systemd user unit"] -->|"auto-start + restart"| A

    subgraph provisioning["Provisioned per machine (gitignored)"]
        B
        M
        SRC["llama.cpp/ source"]
    end

    subgraph toolkit["Toolkit (committed)"]
        SETUP["setup.sh"]
        FETCH["fetch-model.sh"]
        SRV["server.sh"]
        C
    end

    SETUP -->|"detect CPU · fetch release"| B
    FETCH -->|"HF API · verify magic"| M
    SRV -->|"systemctl --user"| S

Request lifecycle

sequenceDiagram
    participant C as Client
    participant S as llama-server
    participant M as Model (GGUF)

    C->>S: POST /v1/chat/completions
    S->>M: tokenize prompt
    M-->>S: prompt tokens
    loop generate tokens
        S->>M: forward pass (CPU)
        M-->>S: next token
    end
    S-->>C: 200 OK + message

Setup flow

flowchart TD
    A["./setup.sh"] --> B{detect platform}
    B -->|OS · arch · CPU caps| C{binaries exist?}
    C -->|yes| D[verify version]
    C -->|no| E[download latest<br/>llama.cpp release]
    D --> F[generate config.sh]
    E --> F
    F --> G[wire launchers<br/>+ env.sh]
    G --> H[install systemd unit]
    H --> I[API ready]

API reference

The server exposes the llama-server endpoints. The ones you'll use:

Endpoint	Purpose
`GET /health`	`{"status":"ok"}` when ready, `503` while loading.
`GET /v1/models`	List loaded model.
`GET /props`	Global generation properties.
`POST /v1/chat/completions`	Chat completion (streaming supported with `stream:true`).
`POST /v1/completions`	Raw completion.
`POST /v1/embeddings`	Embeddings (model-dependent).
`POST /tokenize` / `/detokenize`	Token ↔ text.
`POST /apply-template`	Apply a chat template to messages.

Chat completion (non-streaming)

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role":"system","content":"You are concise."},
                 {"role":"user","content":"What is an API?"}],
    "max_tokens": 256,
    "temperature": 0.7
  }'

Chat completion (streaming)

curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Tell me a story"}],"max_tokens":512,"stream":true}'

cURL → SDK equivalence

Anything that talks OpenAI works unchanged — just point base_url at the local server:

SDK	Code
Python	`OpenAI(base_url="http://127.0.0.1:8080/v1")`
Node.js	`new OpenAI({ baseURL: "http://127.0.0.1:8080/v1" })`
LangChain	`ChatOpenAI(base_url="http://127.0.0.1:8080/v1")`

Benchmarks

Head-to-head on the reference machine (i5-11300H, 4C/8T, AVX-512), using the optimal -t 4 (physical cores).

Model comparison

Model	Size	Params	Prompt (t/s)	Gen (t/s)	Verdict
Llama-3.2-1B Q4_K_M	763 MiB	1.24B	234	21.6	🏆 Fastest generation. Best for high-volume / low-latency.
Qwen2.5-1.5B Q4_K_M	1.04 GiB	1.78B	167	18.7	🏅 Best balance. Smarter than 1B, nearly as fast.
Qwen2.5-3B Q4_K_M	1.95 GiB	3.40B	84	9.7	Strong quality. Noticeably slower.
Qwen3.5-4B Q4_K_M	2.54 GiB	4.21B	40	5.9	Latest generation. Too heavy for snappy CPU chat.

Thread sweep

SMT (Hyper-Threading) hurts on this chip — dropping from 8 threads to 4 physical cores boosted generation 37%:

Threads	Prompt (t/s)	Generation (t/s)
8 (all logical)	70.2	6.8
4 (physical)	82.0	9.3

Rule of thumb: start at your physical core count (lscpu -p | grep -c '^[0-9]*,0'), then sweep ±1 to confirm.

For full methodology and more sweeps (flash-attention, KV cache), see docs/BENCHMARKS.md.

Configuration

All runtime settings live in config.sh (sourced by the service and by env.sh):

Variable	Default	Meaning
`LLAMA_MODEL`	`models/qwen2.5-1.5b-instruct-q4_k_m.gguf`	Path to the GGUF model (relative to `~/llama`).
`LLAMA_PORT`	`8080`	API port.
`LLAMA_HOST`	`127.0.0.1`	Bind address.
`LLAMA_THREADS`	`4`	CPU threads for generation. Tune to your physical core count.
`LLAMA_PARALLEL`	`2`	Concurrent request slots.
`LLAMA_CTX`	`2048`	Context window (tokens).

After editing config.sh, restart:

./server.sh restart

Choosing a model

Trade-off: smaller = faster, larger = smarter. On a CPU, the sweet spot is usually 1–3B parameters. See the benchmark table above.

Use case	Pick
Latency-critical / high throughput	Llama-3.2-1B Q4_K_M (21 t/s)
Daily driver (recommended)	Qwen2.5-1.5B Q4_K_M (18 t/s, much better quality)
Quality over speed	Qwen2.5-3B Q4_K_M (9.7 t/s)
Bleeding edge	Qwen3.5-4B Q4_K_M (5.9 t/s)

RAM budget: keep model size (GiB) under ~40% of your free RAM. On a 7 GB box, stay under ~2.5 GB models for comfort.

Management

./server.sh start      # start (idempotent)
./server.sh stop       # stop
./server.sh restart    # restart
./server.sh status     # show systemd status + health
./server.sh log        # tail the journal

Under the hood these wrap systemctl --user <cmd> llama-server. On macOS (no systemd) the service file is skipped — run the binary directly:

./llama-server -m models/<your-model>.gguf

Project structure

llama/
├── setup.sh                 # bootstrap: detect platform, fetch binaries, wire config
├── fetch-model.sh           # search / download / verify / swap GGUF models
├── server.sh                # start|stop|status|restart|log wrapper
├── config.sh                # all runtime settings (model, port, threads, ctx)
├── env.sh                   # source to put tools on PATH
├── llama-cli                # launcher → bin/llama-cli
├── llama-server             # launcher → bin/llama-server
├── llama-bench              # launcher → bin/llama-bench
├── llama-quantize           # launcher → bin/llama-quantize
├── llama-gguf-split         # launcher → bin/llama-gguf-split
├── assets/                  # diagrams + charts (generated by scripts/)
│   ├── banner.png
│   ├── architecture.svg
│   ├── benchmark-models.png
│   └── benchmark-threads.png
├── scripts/                 # asset generators + docs
│   ├── gen-assets.py
│   ├── gen-architecture.py
│   └── README.md
├── web/                     # zero-dependency chat UI
│   └── index.html
├── examples/                # copy-paste API clients
│   ├── python/chat.py
│   ├── node/chat.mjs
│   └── shell/chat.sh
├── tests/
│   └── integration.sh       # end-to-end server tests
├── docs/
│   ├── QUICKSTART.md
│   ├── BENCHMARKS.md        # full benchmark results + methodology
│   └── ARCHITECTURE.md      # system design + trade-offs
├── bin/                     # llama.cpp binaries (provisioned, gitignored)
├── models/                  # GGUF models (provisioned, gitignored)
├── llama.cpp/               # source checkout (provisioned, gitignored)
└── archives/                # downloaded release tarballs (gitignored)

Requirements

Linux (Ubuntu 20.04+, any distro with systemd) or macOS (13+).
curl, tar, python3 (for model-fetch + asset-generation scripts).
~2 GB RAM free for a 1.5B model; ~3 GB for a 3B model. (Model size in GB ≈ parameter-count × 0.6 for Q4_K_M.)
Network access (first-run only, to fetch binaries + model).

Optional: Docker (20.10+) for containerized runs; matplotlib + pillow to regenerate charts.

No GPU, no CUDA, no Docker required.

Troubleshooting

Symptom	Fix
`{"status":"ok"}` never comes (stuck at 503)	Model still loading — large models take 10–30s. Check `./server.sh log`.
`command not found: llama-cli`	`source ~/llama/env.sh` or use the launcher: `~/llama/llama-cli`.
`failed to load model`	Wrong path in `config.sh`, or corrupt/incomplete GGUF. Re-download with `fetch-model.sh`.
Slow generation	Reduce `LLAMA_THREADS` to physical cores (not SMT). Confirm with `llama-bench`. Close RAM-heavy apps.
`port already in use`	Another llama-server running. `./server.sh stop` or `pkill -f llama-server`.
Service won't auto-start	`systemctl --user enable llama-server.service` and confirm lingering is on: `loginctl enable-linger $USER`.

Documentation

Doc	What's in it
QUICKSTART	Fastest path from clone to first API call.
BENCHMARKS	Head-to-head model comparison, thread sweep, methodology.
ARCHITECTURE	System design, data flow, request lifecycle, trade-offs, security.
CONTRIBUTING	How to improve the toolkit.
CHANGELOG	Version history.
ROADMAP	Planned and considered work.

License

MIT. Do what you want. See LICENSE.

Acknowledgments

ggml-org/llama.cpp — the engine.
Hugging Face — model hub.
Model authors: Meta (Llama), Alibaba (Qwen), and the quantizers (bartowski, MaziyarPanahi, unsloth, hugging-quants).

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
.github		.github
assets		assets
docs		docs
examples		examples
scripts		scripts
tests		tests
web		web
.dockerignore		.dockerignore
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
ROADMAP.md		ROADMAP.md
config.sh		config.sh
docker-compose.yml		docker-compose.yml
env.sh		env.sh
fetch-model.sh		fetch-model.sh
llama-bench		llama-bench
llama-cli		llama-cli
llama-gguf-split		llama-gguf-split
llama-quantize		llama-quantize
llama-server		llama-server
server.sh		server.sh
setup.sh		setup.sh

Folders and files

Latest commit

History

Repository files navigation