Choosing a Model

Choosing a Model

llama-android runs any GGUF model. The hard limits on a phone are RAM and speed, so smaller + more-quantized wins. This page helps you pick.

Quick picks

Model	Size (Q4_K_M)	Good for	Download
Qwen2.5 0.5B	~400 MB	ultra-fast, testing, simple tasks	HF
Qwen2.5 1.5B	~1.0 GB	good balance on low-RAM phones	HF
Gemma 2 2B	~1.6 GB	compact & capable	HF
Llama 3.2 3B	~2.0 GB	best all-round chat	HF
Phi-3.5 Mini 3.8B	~2.2 GB	strong reasoning	HF

Always pick an Instruct / -it (instruction-tuned) variant for chat. Base models only autocomplete and won't follow a system prompt.

RAM budget — the rule of thumb

A loaded model needs roughly:

RAM ≈ model file size + KV cache + ~20% overhead

The KV cache grows with contextSize. As a safe guide:

Phone RAM	Comfortable model ceiling
4 GB	≤ 1.5B (Q4)
6 GB	≤ 3B (Q4)
8 GB	3B–4B (Q4), or 2B at large context
12 GB+	7B–8B (Q4) is possible but slow on CPU

If you exceed available RAM, loadModel throws LlamaException.ModelLoadFailed (often the OS killed the allocation). Drop to a smaller model or lower contextSize.

Quantization cheat-sheet

Quantization trades quality for size/speed. Q4_K_M is the sweet spot most people ship.

Quant	Relative size	Quality	When
Q3_K_M	smallest	lower	very tight RAM
Q4_K_M	small	great	default choice
Q5_K_M	medium	better	if RAM allows
Q6_K / Q8_0	large	near-full	rarely worth it on-device

Where to put the file

GGUF weights are too large for an AAR or the APK's assets. Two options:

1. Download on first launch (recommended): fetch the GGUF over HTTPS into getExternalFilesDir("models") and reuse it. Show a one-time progress bar.
2. Bundle in the APK only for tiny models — it inflates your app download and Play Store has a size cap (use an asset pack for anything sizeable).

val modelFile = File(getExternalFilesDir("models"), "model.gguf")
if (!modelFile.exists()) {
    // download from your CDN / Hugging Face into modelFile, then:
}
val model = Llama.loadModel(modelFile.absolutePath)

Context size vs. speed

Larger contextSize = more memory for history and a slower prompt phase. Start at 2048. Only raise it if you truly need long conversations, and watch RAM.

Embeddings

Any model works with Llama.embed, but dedicated embedding GGUFs (e.g. nomic-embed-text, bge-small) give better vectors for search/RAG and are tiny.

See also Troubleshooting (slow generation, out-of-memory).