Janus — Local LLM Server & OpenAI-Compatible API

Janus is a single Go binary that runs .gguf models on your machine (GPU or CPU) and exposes an OpenAI-compatible API. No Python, no Docker, no Ollama required.

Use it your way: call it from the command line (curl, PowerShell, scripts), wire it into Cursor / Cline / any OpenAI client — same local models, whatever workflow fits you.

• Local inference — llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
• OpenAI-compatible API — /v1/chat/completions, /v1/models
• Hot-swap models — change .gguf without restarting
• Thinking model support — <think> reasoning split into reasoning_content
• Chat template auto-detection — uses the template from GGUF metadata
• Zero dependencies — one .exe on Windows, no Python, no Docker

Disk: plan for the model size (often 2–8 GB per model) plus ~50 MB for Janus + llama.dll.

git clone https://github.com/Vibra-Ingenn/Janus.git
cd Janus
.\build.ps1

build.ps1 downloads pre-built llama.cpp Vulkan DLLs and compiles dist\janus.exe.

Put a .gguf file in the models\ folder. Use the included downloader:

go build -o dist\modelget.exe .\cmd\modelget
.\dist\modelget.exe -repo meta-llama/Llama-3.2-3B-Instruct -file Llama-3.2-3B-Instruct-Q8_0.gguf -out .\models\

Or download any GGUF from Hugging Face.

copy .env.example .env

Edit .env:

INFERENCE_BACKEND=vulkan
JANUS_MODEL_PATH=./models/Llama-3.2-3B-Instruct-Q8_0.gguf
JANUS_MAX_TOKENS=4096

.\dist\janus.exe

Opens http://127.0.0.1:8990 in your browser.

curl http://127.0.0.1:8990/health

git clone https://github.com/Vibra-Ingenn/Janus.git
cd Janus
./build.sh                      # downloads llama.cpp b11146 (Vulkan) into lib/linux, builds dist/janus
cp .env.example .env
# edit .env — set JANUS_MODEL_PATH (and INFERENCE_BACKEND=cpu if no Vulkan)
./dist/janus

• build.sh pins llama.cpp b11146 (the struct layouts in internal/bridge/llama_dl.go match that tag); ./build.sh --version bNNNN tries another one, --skip-download reuses lib/linux/.
• The .so files are copied next to the binary (dist/); they find each other via RUNPATH=$ORIGIN, no LD_LIBRARY_PATH needed.
• Go 1.25+ is required (purego v0.11 is needed to pass llama.cpp's structs by value on Linux). With GOTOOLCHAIN=auto (default) an older go downloads the right toolchain itself.
• JANUS_GPU_LAYERS=-1 (default) offloads every layer; check the log for offloaded N/N layers to GPU.

Supported: linux/amd64; arm64/macOS not tested/not supported.

macOS: build with go build -o dist/janus ./cmd/janus and provide libllama.dylib next to the binary (untested).

curl http://127.0.0.1:8990/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Base URL: http://127.0.0.1:8990/v1

curl http://127.0.0.1:8990/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","stream":true,"messages":[{"role":"user","content":"Tell me a joke"}]}'

Base URL:  http://127.0.0.1:8990/v1
API Key:   (leave blank)

cmd/janus/          Main server (OpenAI-compatible API)
cmd/modelget/       Hugging Face model downloader
internal/engine/    llama.cpp Vulkan/CPU backend
internal/bridge/    DLL loader and FFI bindings
internal/singleton/ Single-instance guard
models/             Put .gguf files here (not committed)
dist/               janus.exe + llama.dll after build

Get-Process -Name "janus" -ErrorAction SilentlyContinue | Stop-Process -Force
go test ./...
go build -o dist\janus.exe .\cmd\janus
.\dist\janus.exe

Or just .\run.ps1. For a full build including llama DLLs, use .\build.ps1.

Contributions welcome — see CONTRIBUTING.md.

MIT — see LICENSE.