Skip to content

2026.9.0

Choose a tag to compare

View all tags
@sockless-coding sockless-coding released this 06 Sep 20:28
· 29 commits to main since this release
2026.9.0
7453ef5

LLM Router — First Release

A lightweight .NET-based routing layer for managing and load-balancing across multiple local LLM inference servers. This is the first public release of LLM Router.

What's included

Multi-protocol API gateway

  • OpenAI-compatible: /v1/chat/completions (streaming + non-streaming), /v1/responses, /v1/models
  • OpenAI Responses API: stateful create / retrieve / delete / cancel with stored conversation state
  • Claude-compatible: /v1/messages
  • Ollama-compatible: /api/chat, /api/generate, /api/tags, /api/show, /api/embed, /api/ps, /api/version
  • Tool / function calling passed through to llama.cpp backends and translated back into Chat Completions or Responses shapes
  • API keys — create and manage keys to secure and meter access, with per-key usage and token-breakdown stats

Routing & supervision

  • Priority-ordered routing rules with round-robin fallback across healthy instances
  • Preset/model affinity — auto-start or restart the server that owns a preset
  • Background health monitoring with automatic routing around failed/crashed backends
  • Resilient backend supervision — a separate wrapper process supervises each llama-server, so the router can restart without killing running backends, and can live-swap models

Engine management

  • Managed llama.cpp builds — download official prebuilt releases from ggml-org/llama.cpp or compile from source using reusable build recipes, with live download/compile progress
  • Track the exact version tag / commit SHA of each build on disk, and check for updates with a commit-by-commit changelog against the latest upstream release
  • Bind servers to a managed build so the router knows precisely which llama.cpp version each backend folder holds
  • Built-in recipe templates plus upstream build-doc references (with parsed cmake configure commands) to help author new recipes

Model management

  • Local GGUF model library with metadata inspection and folder scanning
  • Search & download directly from the Hugging Face Hub with live progress
  • Reusable model presets (sampling, context, GPU/threading, speculative decoding, LoRA, multimodal), optionally linked to a library entry so paths stay in sync
  • Live VRAM usage estimate in the preset editor (including flash-attention impact)

Compute device inventory

  • Automatic detection of available CPU/GPU compute devices with the backends each supports, to help pick the right backend per server

Dashboard & observability

  • Razor Pages web UI for servers, engines, models, presets, devices, request logs, and stats
  • Per-request API log with filtering
  • Throughput, latency, and context-usage charts, plus 24h activity and recent-activity views
  • Per-API-key usage stats with token breakdown
  • Live UI updates via SignalR

Supported backends

llama.cpp (llama-server) across a wide range of compute backends: CPU, CUDA, Vulkan, SYCL, HIP (ROCm), Metal, OpenCL, OpenVINO, MUSA, and CANN.

You can either bring your own llama-server build, or use the built-in Engine management to download official prebuilt releases or compile from source — no manual setup required.

Full Changelog: https://github.com/sockless-coding/llm-router/commits/2026.9.0

Full Changelog: https://github.com/sockless-coding/llm-router/commits/2026.9.0

Assets 4
Loading