Skip to content

2026.9.2

Choose a tag to compare

View all tags
@sockless-coding sockless-coding released this 09 Sep 19:45
· 19 commits to main since this release
2026.9.2
d4b5396

Release 2026.9.2

New Features

Live context (KV-cache) usage tracking & context-aware queuing

  • Poll llama.cpp's /slots endpoint to track per-slot context occupancy, surfaced live (via SignalR) on the dashboard and server detail pages as a "Context used" bar (turns red at ≥ 90%).
  • Optional context-aware queuing: when enabled, new requests are held in the queue while a server's busiest slot is at/above a configurable KV-cache threshold and it already has work in flight, so in-flight turns can finish and free context.
  • New gateway settings: ContextAwareQueuing toggle and ContextUsageQueueThresholdPercent, with Settings UI.
  • Added LlamaRuntimeUsage model and LlamaSlotsSnapshot parser, plus unit tests for slot parsing and capacity-hold logic.

Configurable local/UTC timestamp display

  • New server-rendered Html.Timestamp(...) helper emits <time datetime="…Z"> elements holding UTC values; a layout script rewrites the visible text into the viewer's chosen zone (local by default), persisted like the theme.
  • Added a timestamp zone selector (Local/UTC) to the settings panel.
  • Handles htmx:afterSettle so hx-boost swaps keep timestamps updated; tooltips show both UTC and local (ms precision).
  • Replaced inline UTC formatting across the dashboard, request log, servers, API keys, and engines changelog/releases pages.

Improvements

Request-log performance

  • ApiRequestLog timestamps are now stored as round-trip UTC TEXT (lexicographically sortable), so time filters, ORDER BY, and retention cutoffs translate to SQL and ride IX_ApiRequestLogs_Timestamp instead of filtering client-side.
  • GetRecentLogsAsync, DeleteOlderThanAsync, and GetSummaryStatsAsync now use indexed SQL-side filtering/aggregation.
  • RetentionCleanupService now runs VACUUM after a purge to compact freed payload pages; removed a duplicate service registration.
  • Added a migration and tests for the new value conversion.

Graceful backend error handling

  • New BackendErrorClassifier detects routine, user-actionable backend failures (llama.cpp context-exceeded) and surfaces them as clean, well-formed client errors instead of unhandled exceptions that sever the connection:
    • OpenAI: mid-stream → in-band SSE error frame + [DONE]; non-streaming → structured error with 400/502.
    • Ollama: mid-stream → NDJSON error line; non-streaming → { error } JSON with 400/502.
    • Responses: streaming → response.failed event; non-streaming → persists a failed row and returns a failed response object.
  • LlamaCppProvider logs context-exceeded failures at Warning (not Error), since they aren't provider malfunctions.

Fixes

  • fix(api): ensure all non-assistant messages (system, user, tool, …) carry an empty content field before forwarding to llama.cpp, preventing 400s when replaying stored conversations with a null-content turn.

Full Changelog: 2026.9.1...2026.9.2

Assets 4
Loading