Skip to content

OpenAI Endpoint Guide

Jump to bottom
Carlos Catly edited this page Jul 25, 2026 · 3 revisions

Setting Up an OpenAI-Compatible Endpoint (Ollama or llama.cpp)

This guide walks you through setting up a local translation LLM (such as Gemma 4) on your computer and configuring it as an OpenAI-Compatible Endpoint in Aibou.

You can choose either Ollama (recommended for simplicity) or llama.cpp (for power users who manage raw GGUF model files manually).

Why a Custom System Prompt is Required

When Aibou sends translation requests—especially batch translations—it uses a specific delimiter (|||) to separate multiple lines of text in a single request.

Without explicit instructions, general chat AI models may answer conversationally (e.g., "Here is your translation: ...") or strip out required formatting delimiters. Whether using Ollama or llama.cpp, we must provide a custom System Prompt instructing the model to act strictly as an automated translation engine.

The required system prompt is:

You are a direct professional Japanese-to-English translation engine.
Your task is to translate the text sent by the user directly into fluent English.

Strict Rules:
1. ONLY output the direct English translation. Do NOT include explanations, greetings, introduction, or conversational filler of any kind.
2. If the input text contains multiple segments separated by the '|||' delimiter, you MUST preserve the exact same '|||' delimiter between your translated segments in the output. Do not add spaces or formatting around '|||'.
3. Maintain the exact same number of separated segments as provided in the input.

Option 1: Using Ollama (Recommended)

1. Install Ollama & Create Modelfile

  1. Download and install Ollama.
  2. Create a text file named Modelfile (no extension) anywhere on your computer with the following content: 
    FROM gemma4

SYSTEM """
You are a direct professional Japanese-to-English translation engine.
Your task is to translate the text sent by the user directly into fluent English.

Strict Rules:
1. ONLY output the direct English translation. Do NOT include explanations, greetings, introduction, or conversational filler of any kind.
2. If the input text contains multiple segments separated by the '|||' delimiter, you MUST preserve the exact same '|||' delimiter between your translated segments in the output. Do not add spaces or formatting around '|||'.
3. Maintain the exact same number of separated segments as provided in the input.
"""
  3. Open a terminal in the folder containing your Modelfile and run this command to download Gemma 4 and build your translator model: 
    ollama create aibou-gemma4 -f Modelfile

2. Enable Network Access for Ollama

By default, Ollama only listens on localhost (127.0.0.1). To allow your phone to connect over Wi-Fi or Tailscale:

  • Windows (PowerShell): 
    $env:OLLAMA_HOST="0.0.0.0"
ollama serve
    (Tip: To make this permanent on Windows, add OLLAMA_HOST with value 0.0.0.0 under System Environment Variables and restart Ollama).
  • macOS / Linux: 
    OLLAMA_HOST=0.0.0.0 ollama serve

Option 2: Using llama.cpp (llama-server)

If you prefer downloading standalone .gguf model weights and running llama.cpp, you can use its built-in tool called llama-server, which natively exposes an OpenAI-compatible API endpoint!

1. Save the System Prompt File

Create a plain text file named system_prompt.txt inside your llama.cpp directory and paste the required translation instructions into it:

You are a direct professional Japanese-to-English translation engine.
Your task is to translate the text sent by the user directly into fluent English.

Strict Rules:
1. ONLY output the direct English translation. Do NOT include explanations, greetings, introduction, or conversational filler of any kind.
2. If the input text contains multiple segments separated by the '|||' delimiter, you MUST preserve the exact same '|||' delimiter between your translated segments in the output. Do not add spaces or formatting around '|||'.
3. Maintain the exact same number of separated segments as provided in the input.

2. Launch llama-server

Open your terminal inside the llama.cpp folder and start the HTTP server using --host 0.0.0.0 (to enable local Wi-Fi and network access), pointing to both your GGUF file and your system prompt file:

# On Windows
llama-server.exe --host 0.0.0.0 --port 8080 -m models\gemma-4.gguf --system-prompt-file system_prompt.txt -c 4096

# On Linux / macOS
./llama-server --host 0.0.0.0 --port 8080 -m models/gemma-4.gguf --system-prompt-file system_prompt.txt -c 4096

(Note: Adjust models\gemma-4.gguf to match the real file path of your GGUF file. By default, llama-server listens on port 8080).

Find Your PC's Local IP Address

Your Android phone needs your PC's Wi-Fi IP address to communicate with your server:

  • Windows: Open PowerShell / Command Prompt and run ipconfig. Look for your active adapter's IPv4 Address (e.g., 192.168.1.50).
  • macOS/Linux: Run ifconfig or ip a.

Configure Aibou

Open the Aibou App on your Android device and navigate to your API configurations:

  1. Select OpenAI Endpoint as your translation provider.
  2. Enter your server details:
    • API URL:
      • If using Ollama: http://<YOUR-PC-IP>:11434/v1/chat/completions
        (Example: http://192.168.1.50:11434/v1/chat/completions)
      • If using llama.cpp (llama-server): http://<YOUR-PC-IP>:8080/v1/chat/completions
        (Example: http://192.168.1.50:8080/v1/chat/completions)
    • API Key: Enter any placeholder string such as local or ollama
      (Both Ollama and local llama-server ignore API keys by default, but OpenAI client libraries require a non-empty string here).
    • Model Name:
      • If using Ollama: aibou-gemma4
      • If using llama.cpp: gemma-4 (Any text works here; llama-server automatically uses whichever GGUF file is currently running).
  3. Tap Save / Test to activate the endpoint!

Optional: Access Outside Your Local Network (Tailscale)

If you want to use Aibou with your local translation model while away from home (on mobile data or public Wi-Fi) without risking security by port forwarding your router, you can use Tailscale:

  1. Install the Tailscale app on both your PC running the LLM server and your Android device running Aibou.
  2. Log into the same Tailscale account on both devices to connect them to your private virtual mesh network (Tailnet).
  3. Find your PC's assigned Tailscale IP address in the Tailscale app (usually starting with 100.x.x.x).
  4. In Aibou, replace your home Wi-Fi IP in the API URL with your PC's Tailscale IP address:
    • Ollama via Tailscale: http://100.85.23.41:11434/v1/chat/completions
    • llama.cpp via Tailscale: http://100.85.23.41:8080/v1/chat/completions

As long as Tailscale is active on both devices and your local server was started listening on 0.0.0.0, Aibou can securely translate using your PC's LLM from anywhere in the world!

Clone this wiki locally