Skip to content

2026.11.3

Latest

Choose a tag to compare

@sockless-coding sockless-coding released this 14 Nov 09:55
· 1 commit to main since this release
2ebd17f

LLM Router 2026.11.3

This release adds router-wide SYCL keep-alive to keep models loaded on headless Intel GPUs, hourly per-API-key usage tracking with reports, and a fix so the playground preset picker defaults to the model that's actually loaded.

SYCL keep-alive for headless Intel GPUs

Headless Intel GPUs (no monitor attached) can have the driver drop VRAM, silently evicting loaded models. The router can now supervise a keep-alive process that prevents this.

  • New bundled sycl-keepalive tool (source + prebuilt exe) shipped with the app.
  • New SYCL Keep-alive settings (Off / Headless / Always) with a keep-alive section on the Devices page: mode, oneAPI setup command, executable path override, and save/stop/start/re-check actions. Disabling an enabled SYCL GPU releases the keep-alive on it first.
  • Headless detection walks the device tree per GPU (works in session 0) to find attached monitors and PCI addresses, so the keep-alive targets only headless cards.
  • The decision logic lives in a pure, unit-tested SyclKeepAlivePlanner; a hosted service starts/stops/rechecks the process each minute and exposes status.

Shared ownership across router installs

  • Keep-alive ownership is recorded machine-wide under %ProgramData%\LLM Router (PID, start time, config, install path), so the next router to run — e.g. a dotnet run build after the service — takes the keep-alive over to keep, retarget, or stop it instead of starting a second one.
  • A keep-alive running as another user (e.g. SYSTEM vs a dev build) is treated as running externally rather than failing to stop and doubling up; the page surfaces a "run elevated to take it over" state when takeover can't inspect the process.
  • The keep-alive intentionally outlives the router, so it now runs from a staged copy in %ProgramData%\LLM Router\sycl-keepalive\<hash>\ — named by content hash so a new build gets a new path and the old process is replaced, and stale copies are pruned on start. This stops the executable from locking the install folder during rebuilds/updates. Keep-alives started from an older install-folder copy are still recognized and taken over.

Hourly per-API-key usage tracking & reports

  • New ApiKeyUsageHourly rollup: prompt and completion token counts recorded per API key, model, and hour as requests flow through the router.
  • Model attribution is normalized across all API gateways (embeddings, images, passthrough, Ollama, System One) so rollups use the resolved preset name.
  • New Usage view on the API keys page plus an /api/stats/usage-report endpoint, linked from the Dashboard.

Playground: preset picker defaults to the loaded model

  • A remembered or previously selected preset could point at a model that's no longer running, making the first send silently swap models. The playground pages now report which presets have running servers, and the picker prefers a loaded preset over the remembered one — unless an explicit ?presetId= was given.

What's Changed

Full Changelog: 2026.11.2...2026.11.3