Yes, you can run local LLMs on your Android phone — completely offline — using llama.cpp in Termux!
This guide walks you step by step through compiling llama.cpp, downloading quantized .gguf models, running TinyLLaMA or LLaMA 2 7B, and even setting up a simple Chat UI.
- What You’ll Need
- Installing Termux and Required Packages
- Cloning and Building llama.cpp
- Downloading TinyLLaMA and LLaMA 2 7B Quantized Models
- Running the Model in CLI
- Setting Up a Basic Web Chat UI
- Final Thoughts & Tips
- A relatively recent Android phone
- Minimum 4 GB RAM (8 GB or more for LLaMA 2 7B)
- Minimum 4-core CPU (Snapdragon 7xx or 8xx preferred)
- Termux (Download from F-Droid, not Play Store)
- At least:
- ~500MB for TinyLLaMA
- ~4–5GB for LLaMA 7B quantized
Open Termux and run the following:
pkg update && pkg upgrade -y
pkg install git cmake clang build-essential wget python -yMake sure you have storage access enabled:
termux-setup-storageThis allows file access from ~/storage for downloaded models.
# Clone the llama.cpp repo
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
# Compile the main binary
makeThis builds the ./main program that runs GGUF models locally.
TinyLLaMA is super lightweight and works well on mid-range phones.
Download command:
mkdir -p models/tinyllama
cd models/tinyllama
# Download the quantized GGUF model
wget https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf
cd ../../If your phone is powerful (Snapdragon 8 Gen 1+, or 12GB RAM), try this:
mkdir -p models/llama2
cd models/llama2
# Download a 7B quantized model (Q4_0 = decent quality, smaller size)
wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_0.gguf
cd ../../You’re now ready to run the model using the CLI!
./main -m models/tinyllama/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf -p "Hello! Who are you?"./main -m models/llama2/llama-2-7b-chat.Q4_0.gguf -p "What is the capital of France?"./main -m models/tinyllama/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf -iYou can now type prompts interactively.
You can build a simple web-based chat UI running locally on your Android phone.
pip install flask flask-corsnano chat_server.pyPaste the following:
from flask import Flask, request, jsonify
import subprocess
app = Flask(__name__)
@app.route('/chat', methods=['POST'])
def chat():
prompt = request.json.get("prompt", "")
if not prompt:
return jsonify({"error": "No prompt given."}), 400
# Call llama.cpp with the prompt
result = subprocess.run(
["./main", "-m", "models/tinyllama/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf", "-p", prompt],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True
)
return jsonify({"response": result.stdout})
if __name__ == '__main__':
app.run(host='0.0.0.0', port=7860)python chat_server.pyNow go to your browser and POST to:
http://localhost:7860/chat
Or use an app like Postman to test it.
Make a simple chat page (chat.html) like:
<!DOCTYPE html>
<html>
<head>
<title>TinyLLaMA Chat</title>
</head>
<body>
<h2>Chat with TinyLLaMA</h2>
<textarea id="input" rows="4" cols="50"></textarea><br>
<button onclick="send()">Send</button>
<pre id="response"></pre>
<script>
function send() {
fetch("http://localhost:7860/chat", {
method: "POST",
headers: {"Content-Type": "application/json"},
body: JSON.stringify({prompt: document.getElementById("input").value})
})
.then(res => res.json())
.then(data => {
document.getElementById("response").innerText = data.response;
});
}
</script>
</body>
</html>Open it in a browser and chat locally!
- Use TinyLLaMA for best performance on most phones.
- Use
-n 256to control number of output tokens (example:-n 256 -p "Explain gravity"). - Monitor RAM usage — LLaMA 7B needs around 4–6 GB RAM for Q4.
- You can create wrapper scripts or integrate with Android web UIs using Termux:API if you want!
Running local LLMs on Android is now a reality with llama.cpp, especially with small models like TinyLLaMA. For larger models like LLaMA 7B, high-end phones or tablets work well — and with Flask, you can even build your own mobile ChatGPT clone completely offline.