2026.10.0

Release 2026.10.0

New Features

Strata engine support

• Added first-class support for Strata, a MoE-offloading engine with a Python HTTP frontend, alongside llama.cpp
• Generalized the engine build system so installs are tracked per-engine (EngineBuild/EngineBuilds with a ServerEngine column); servers now bind to same-engine installs
• New engine discovery abstractions (IEngineDescriptor, IEngineCatalog, IEngineInstallHandler) with an EngineCatalog that resolves registered engines
• Strata install/update jobs: side-by-side release installs (clone tag, .venv, unattended setup), in-place updates (git pull --ff-only + setup.py --update), engine rollback, and stop-tracking without deleting files
• Split LlamaCppProvider into a shared ManagedServerProviderBase plus engine-specific providers; the new StrataProvider runs Strata's server.py on the router-assigned port, with /props and /slots mirroring llama.cpp so slot gating and KV usage work unchanged
• On first preset start, the model is prepared with Strata's setup.py in existing-GGUF mode (build pack, download MTP draft, write a per-PC run config), with progress streamed to the server status
• Per-engine launch options are stored on ModelPreset via a new EngineSettings JSON column (keyed <engine>.<setting>), so engines can add their own options without a column per option
• Restructured the Engines page into per-engine tabs; server create/edit pages now offer engine selection and engine-appropriate install folder fields
• Preset form gains a Strata section: context length, GPU, images, KV cache type, low-RAM mode, and parallel requests (blank = Strata's recommendation)

Split GGUF support

• New SplitGguf helper parses -NNNNN-of-MMMMM filenames; the model library lists a split model once (first shard, total size) and delete removes every shard
• ModelDownloadService downloads all shards in sequence, pinned to the resolved revision, with combined progress
• The Hugging Face picker shows split models as a single entry with a parts count

Routing: model aliases & fallback presets

• Model aliases with * wildcards (e.g. gpt-4o, claude-*) so clients with hardcoded model ids resolve to a preset (new ModelAliasMatcher)
• Per-preset fallback chain: serves when the primary's server is errored, and takes overflow when the primary is at capacity and the fallback is already loaded; request stats/logs are retargeted to the preset that actually served

Memory management

• Servers sharing a GPU form a memory group with a VRAM budget; before a model loads, least-recently-used idle servers in the group are stopped until the estimated footprint fits (MemoryGroupScheduler + PresetMemoryEstimator)
• Per-server idle unload after N minutes (IdleUnloadService)
• Preset "keep loaded" flag exempts a server from automatic unloading

Embeddings

• OpenAI POST /v1/embeddings and Ollama POST /api/embed / POST /api/embeddings, forwarded verbatim via a new provider SendRawRequestAsync + RouteRequest.BackendEndpoint

Playground

• New dashboard feature for chatting with any preset and comparing runs under different launch settings
• Chat page with per-reply timings (TTFT, prompt/generation tok/s, cached tokens, speculative-decoding draft acceptance), run labels, Repeat xN, and CSV export
• Run history page: runs persist to a new PlaygroundRuns table with a snapshot of the preset name and launch settings (hash of the generated command line), grouped by preset/label/settings; unaffected by request-log retention and soft-deleted when the preset is removed
• Parse llama.cpp draft_n / draft_n_accepted into LlamaCppTimings and backfill them in OpenAiHandler when only scalar response values exist

Servers

• Added a Delete button to the server list with a confirmation dialog and CSRF-protected POST; the new OnPostDeleteAsync 404s unknown servers and 409s when a server is mid-transition (starting/stopping/reconnecting), and drops the server's memory group once no other instance references it

Improvements

• Service install script — install-service.ps1 now defaults -PublishDir to $PSScriptRoot instead of a hardcoded bin/Release/.../publish path, so the script locates the executable relative to itself
• Release asset verification — GitHub release assets now carry a Digest field (SHA-256) used to verify downloaded engine assets against GitHub's published digest
• Engine variant selection — release installs now pick an engine build (CUDA, CUDA 12, or AMD/HIP) and download the matching verified asset

Fixes

• Engine build commit capture — when a source build ran under an env setup command (e.g. oneAPI setvars), the command's banner was recorded as the build's commit SHA. CaptureGitAsync now returns only the last non-empty output line, and a reconcile pass sanitizes stored CommitSha values (extracting the trailing 40-char hex SHA, or clearing the value if none is found) so stale rows self-heal

What's Changed

• Add Strata engine support and generalize engine build management by @sockless-coding in #2
• Add Playground: preset chat with per-reply stats and benchmark run history by @sockless-coding in #3
• Add model aliases & fallback presets, memory groups & idle unload, and embeddings by @sockless-coding in #4
• Add delete server action to servers page by @sockless-coding in #5

New Contributors

• @sockless-coding made their first contribution in #2

Full Changelog: 2026.9.3...2026.10.0

2026.9.3

New Features

• Massively expanded llama.cpp presets — ~70 new server flags added to presets, including endpoint toggles, model alias/tags, video input, LoRA init-without-apply, CORS/SSL settings, and logging options. The preset form now uses a tabbed layout with a live generated CLI-args preview.
• Draft models & mmproj projector links — Presets can link SpecDraftModelId and MmprojId to registry entries, so draft models and multimodal projectors can be picked from the model library and stay in sync with LocalModel.FilePath, mirroring the main model link.
• Models grouped by repo — The Models index now collapses multiple quants and auxiliary files of the same model into a single expandable group row with aggregate status, size, and preset-usage totals. GGUF files are classified as primary models vs. mmproj projectors, with projectors shown as "extra" files under their group.
• Quick-add links on the Models page —
  • "+ Version" on Hugging Face groups jumps straight to that repo's file list (?repoId=) instead of requiring a search.
  • "+ File" scopes the create-page .gguf candidate scan to the model's own folder (?folder=).
  • "Model Card" link to the HF repo, plus "View on Hugging Face" on the model details page.
• In-place model updates — Hugging Face models can be re-downloaded over their existing path, updating the same registry row so presets referencing the model keep working. The models page has an "update" action with a confirmation prompt (warns when presets use the model).

Fixes

• Forced non-streaming responses in queued handler — The non-streaming request path now explicitly sets stream=false, so queued payloads built for streaming requests are no longer answered with SSE framing that can't be parsed as a single JSON document.
• Preset update integrity — Hand-maintained field copies in PresetManager.UpdateAsync were replaced with CurrentValues.SetValues so newly added fields can't be silently dropped on update.
• Culture-independent decimal binding — Invariant culture is now forced on all threads so decimal preset fields bind correctly on machines using a comma decimal separator.

Full Changelog: 2026.9.2...2026.9.3

2026.9.2

Release 2026.9.2

New Features

Live context (KV-cache) usage tracking & context-aware queuing

• Poll llama.cpp's /slots endpoint to track per-slot context occupancy, surfaced live (via SignalR) on the dashboard and server detail pages as a "Context used" bar (turns red at ≥ 90%).
• Optional context-aware queuing: when enabled, new requests are held in the queue while a server's busiest slot is at/above a configurable KV-cache threshold and it already has work in flight, so in-flight turns can finish and free context.
• New gateway settings: ContextAwareQueuing toggle and ContextUsageQueueThresholdPercent, with Settings UI.
• Added LlamaRuntimeUsage model and LlamaSlotsSnapshot parser, plus unit tests for slot parsing and capacity-hold logic.

Configurable local/UTC timestamp display

• New server-rendered Html.Timestamp(...) helper emits <time datetime="…Z"> elements holding UTC values; a layout script rewrites the visible text into the viewer's chosen zone (local by default), persisted like the theme.
• Added a timestamp zone selector (Local/UTC) to the settings panel.
• Handles htmx:afterSettle so hx-boost swaps keep timestamps updated; tooltips show both UTC and local (ms precision).
• Replaced inline UTC formatting across the dashboard, request log, servers, API keys, and engines changelog/releases pages.

Improvements

Request-log performance

• ApiRequestLog timestamps are now stored as round-trip UTC TEXT (lexicographically sortable), so time filters, ORDER BY, and retention cutoffs translate to SQL and ride IX_ApiRequestLogs_Timestamp instead of filtering client-side.
• GetRecentLogsAsync, DeleteOlderThanAsync, and GetSummaryStatsAsync now use indexed SQL-side filtering/aggregation.
• RetentionCleanupService now runs VACUUM after a purge to compact freed payload pages; removed a duplicate service registration.
• Added a migration and tests for the new value conversion.

Graceful backend error handling

• New BackendErrorClassifier detects routine, user-actionable backend failures (llama.cpp context-exceeded) and surfaces them as clean, well-formed client errors instead of unhandled exceptions that sever the connection:
  • OpenAI: mid-stream → in-band SSE error frame + [DONE]; non-streaming → structured error with 400/502.
  • Ollama: mid-stream → NDJSON error line; non-streaming → { error } JSON with 400/502.
  • Responses: streaming → response.failed event; non-streaming → persists a failed row and returns a failed response object.
• LlamaCppProvider logs context-exceeded failures at Warning (not Error), since they aren't provider malfunctions.

Fixes

• fix(api): ensure all non-assistant messages (system, user, tool, …) carry an empty content field before forwarding to llama.cpp, preventing 400s when replaying stored conversations with a null-content turn.

Full Changelog: 2026.9.1...2026.9.2

2026.9.1

What's Changed

New Features

• Per-server concurrency limiting — The router now caps concurrent requests to each llama.cpp server at its actual slot count. Requests queue when all candidate servers are saturated.
• Live slot usage and expanded telemetry — Added llama.cpp /props telemetry, including slots, context size, vision/audio support, model path, and build information. Server detail pages and the dashboard now show live slot usage.
• Configurable default parallel slots — Added GatewaySettings.DefaultParallelSlots, defaulting to 4 when server capacity and preset settings are unavailable.

Bug Fixes

• Tool-call streaming — Fixed malformed tool calls during speculative/MTP decoding by buffering nameless tool-call fragments and emitting the correct OpenAI wire format.
• Ollama vision detection — Models configured with an mmproj path or URL are now correctly reported as vision-capable.
• Ollama context reporting — /api/tags and /api/show now report the preset's configured context size, falling back to the model's native GGUF value when unset.
• GPU registry handling — Unreadable GPU driver registry subkeys no longer crash the Devices page; inaccessible entries are skipped gracefully.

Full Changelog: 2026.9.0...2026.9.1

Full Changelog: 2026.9.0...2026.9.1

2026.9.0

LLM Router — First Release

A lightweight .NET-based routing layer for managing and load-balancing across multiple local LLM inference servers. This is the first public release of LLM Router.

What's included

Multi-protocol API gateway

• OpenAI-compatible: /v1/chat/completions (streaming + non-streaming), /v1/responses, /v1/models
• OpenAI Responses API: stateful create / retrieve / delete / cancel with stored conversation state
• Claude-compatible: /v1/messages
• Ollama-compatible: /api/chat, /api/generate, /api/tags, /api/show, /api/embed, /api/ps, /api/version
• Tool / function calling passed through to llama.cpp backends and translated back into Chat Completions or Responses shapes
• API keys — create and manage keys to secure and meter access, with per-key usage and token-breakdown stats

Routing & supervision

• Priority-ordered routing rules with round-robin fallback across healthy instances
• Preset/model affinity — auto-start or restart the server that owns a preset
• Background health monitoring with automatic routing around failed/crashed backends
• Resilient backend supervision — a separate wrapper process supervises each llama-server, so the router can restart without killing running backends, and can live-swap models

Engine management

• Managed llama.cpp builds — download official prebuilt releases from ggml-org/llama.cpp or compile from source using reusable build recipes, with live download/compile progress
• Track the exact version tag / commit SHA of each build on disk, and check for updates with a commit-by-commit changelog against the latest upstream release
• Bind servers to a managed build so the router knows precisely which llama.cpp version each backend folder holds
• Built-in recipe templates plus upstream build-doc references (with parsed cmake configure commands) to help author new recipes

Model management

• Local GGUF model library with metadata inspection and folder scanning
• Search & download directly from the Hugging Face Hub with live progress
• Reusable model presets (sampling, context, GPU/threading, speculative decoding, LoRA, multimodal), optionally linked to a library entry so paths stay in sync
• Live VRAM usage estimate in the preset editor (including flash-attention impact)

Compute device inventory

• Automatic detection of available CPU/GPU compute devices with the backends each supports, to help pick the right backend per server

Dashboard & observability

• Razor Pages web UI for servers, engines, models, presets, devices, request logs, and stats
• Per-request API log with filtering
• Throughput, latency, and context-usage charts, plus 24h activity and recent-activity views
• Per-API-key usage stats with token breakdown
• Live UI updates via SignalR

Supported backends

llama.cpp (llama-server) across a wide range of compute backends: CPU, CUDA, Vulkan, SYCL, HIP (ROCm), Metal, OpenCL, OpenVINO, MUSA, and CANN.

You can either bring your own llama-server build, or use the built-in Engine management to download official prebuilt releases or compile from source — no manual setup required.

Full Changelog: https://github.com/sockless-coding/llm-router/commits/2026.9.0

Full Changelog: https://github.com/sockless-coding/llm-router/commits/2026.9.0