🚀 LiteRT API Server

⚠️ Disclaimer (Vibe Coder Edition):
This project does not claim to be "professional" or ready for enterprise production! It is above all a "home project" created out of pure passion for AI, coded "on the vibe" (vibe coder) to have fun with local LLMs on Android. If the code isn't perfectly academic, that's normal—it's made with love! ❤️

Note: This API server is primarily developed, tested, and optimized for the Gemma model family (Gemma 2B, 4B, etc.) in .litertlm or .tflite format. While other models may theoretically load, the internal prompt formatting and memory management are specifically tailored for Gemma.

LiteRT API Server is a powerful Android-based server that exposes Google's LiteRT (formerly TensorFlow Lite / MediaPipe) as a high-performance, local REST API.

Built in Kotlin with Ktor, this application turns any compatible Android device (Phone, Tablet, or TV Box) into an autonomous AI server running Large Language Models (LLMs) like Gemma entirely on-device (Edge AI).

---

✨ Features

• 🧠 Local LLM Inference: Run .tflite / .litertlm models directly on the device using the CPU, GPU, or NPU.
• 🌐 REST API: Exposes OpenAI-compatible /v1/chat/completions endpoints.
• 🛠️ Built-in Web Admin Panel: Manage models, upload .tflite files directly from your browser, and monitor hardware metrics.
• 📊 Real-time Monitoring: Tracks RAM usage and device temperature to prevent thermal throttling.
• ⚙️ Dynamic Parameters: Supports modifying temperature, topK, topP, and system prompts per request.
• 🛡️ Secure Access: API Key (Bearer Token) support to secure your endpoints.

---

📸 Web Administration Panel

The server runs a local web interface accessible via browser on your network (e.g., http://<ANDROID_IP>:3000/admin). From the dashboard, you can:

1. Upload Base Models (.tflite).
2. Manage active inferences.
3. Restart the background service.

---

🛠️ Tech Stack

• Language: Kotlin
• Server Framework: Ktor (Embedded Server)
• AI Engine: Google LiteRT (MediaPipe / TFLite)
• Frontend (Admin Panel): Vanilla HTML/CSS/JS with a dark mode glassmorphism UI.

---

🚀 Getting Started

Prerequisites

• Android Studio.
• An Android device running Android 10+ (a powerful GPU is recommended for LLM inference).
• Pre-converted .tflite or .litertlm models (e.g., Gemma 2B converted for LiteRT).

Installation

1. Clone this repository.
2. Open the project in Android Studio.
3. Build and install the APK on your device.
4. Launch the app and grant the necessary permissions (Storage & Network).
5. The server will start automatically on port 3000.

---

📖 API Usage Example

You can query the server locally using standard curl or any OpenAI-compatible client.

curl -X POST http://<ANDROID_IP>:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello!"}
    ],
    "temperature": 0.7,
    "topK": 40
  }'

---

🔒 Security Notice

This application is designed as a Homelab / Hobby project to be run on a trusted local network.

Important: The Web Administration Panel (/admin) is not authenticated. Anyone on your local network can access it, upload models, or restart the service.

If you plan to expose this API Server to the public internet or an untrusted network, DO NOT expose port 3000 directly. It is highly recommended to place the application behind a secure reverse proxy (like Nginx, Caddy, or Traefik), configure SSL/TLS, and add Basic Authentication (or Authelia) to protect the /admin routes.

---

📜 License

This project is licensed under the MIT License.

---

Made with ❤️ by Clement Monteiro