Quick Start

Quick Start — your first on-device chat

This walkthrough takes a complete beginner from an empty Activity to a working chat completion running entirely on the phone. No server, no API key.

Step 0 — Add the dependency

See Installation. In short:

implementation("dev.ffmpegkit-maintained:llama-android:0.1.0")

Step 1 — Get a GGUF model onto the device

LLM weights are hundreds of MB to several GB — far too big to bundle inside an AAR. You ship or download a GGUF file. For a first test, grab a tiny one:

Qwen2.5 0.5B Instruct (Q4_K_M, ~400 MB) — download from Hugging Face.

For local testing you can push it straight to your app's external files dir with adb:

adb push qwen2.5-0.5b-instruct-q4_k_m.gguf \
  /sdcard/Android/data/<your.app.id>/files/models/model.gguf

In production you typically download it on first launch and cache it in getExternalFilesDir("models"). See Choosing a Model for what to ship.

Step 2 — Load, complete, release

Every heavy call is a suspend function, so run them from a coroutine (lifecycleScope, a ViewModel scope, etc.).

import dev.ffmpegkit.llama.Llama
import dev.ffmpegkit.llama.LlamaConfig
import androidx.lifecycle.lifecycleScope
import kotlinx.coroutines.launch
import java.io.File

lifecycleScope.launch {
    val modelPath = File(getExternalFilesDir("models"), "model.gguf").absolutePath

    // 1. Load the model + create its inference context.
    val model = Llama.loadModel(
        modelPath = modelPath,
        config = LlamaConfig(contextSize = 2048, threads = 4),
    )

    // 2. Ask something. The model's own chat template is applied automatically.
    val result = Llama.complete(
        model,
        prompt = "Explain gravity to a 5-year-old.",
        systemPrompt = "You are a friendly teacher.",
        maxTokens = 256,
    )

    println(result.text)
    println("${result.tokensGenerated} tokens · ${result.tokensPerSecond} tok/s")

    // 3. ALWAYS free the native memory when you're done with the model.
    Llama.releaseModel(model)
}

That's a complete chatbot turn. systemPrompt and maxTokens are optional.

Step 3 — Understand the config

LlamaConfig controls the model and sampling:

Property	Default	Meaning
contextSize	2048	context window in tokens (prompt + reply).
threads	4	CPU threads for inference.
gpuLayers	0	GPU-offloaded layers. 0 = CPU only. GPU (Vulkan) is a Pro feature.
temperature	0.7f	randomness; <= 0 = deterministic (greedy).
topP	0.9f	nucleus sampling.
topK	40	top-k sampling.
seed	-1	RNG seed; -1 = random each run.

For reproducible output set temperature = 0f (greedy) or fix seed.

Step 4 — The result object

Llama.complete() returns a LlamaResult:

Field	Type	Meaning
text	String	the generated reply.
tokensGenerated	Int	number of tokens produced.
tokensPerSecond	Float	generation throughput.
promptEvalTimeMs	Long	time spent ingesting the prompt.
generateTimeMs	Long	time spent generating.

Step 5 — Embeddings (optional)

Turn text into a vector for semantic search, RAG, or clustering:

val vector: FloatArray = Llama.embed(model, "on-device AI")
// Compare two vectors with cosine similarity for "how related are these texts?"

Lifecycle rules (read these)

• One LlamaModel is NOT thread-safe. Don't call complete on the same model from two coroutines at once — serialize them. (Concurrent per-session KV caches are a Pro feature; see Free vs Pro.)
• Always releaseModel. The model holds hundreds of MB of native RAM the garbage collector cannot see. Release it in onCleared() / onDestroy() or as soon as you're done. Calling it twice is safe.
• Loading is expensive (reads the whole file, allocates the KV cache). Load once, reuse the model for many complete calls, release at the end.

Next: Choosing a Model · Troubleshooting.