v0.9.3 - Performance and Runtime Stabilization

KeepRoLLMing v0.9.3

KeepRoLLMing 0.9.3 is a performance and runtime-stabilization release. It
focuses on making long-lived, concurrent streams more dependable while giving
operators clearer visibility and safer controls around the proxy boundary.

Highlights

• More dependable long-running streams

  • A bounded shared upstream HTTP transport makes connection-pool behaviour
    explicit and observable.
  • Downstream cancellation now promptly closes the upstream stream, preventing
    backend generations from continuing after the client has gone away.
  • Graceful shutdown cancels background config watching and closes the shared
    HTTP client before observability workers stop.

• Observability that stays out of the request path

  • PLAIN, JSON, compact-server, and performance projections use independent
    bounded workers, so slow filesystem output cannot stall a response stream.
  • Lifecycle diagnostics cover downstream delivery failures, cancellation, and
    upstream request transitions.
  • Streaming progress now accounts for reasoning and tool-call output, while
    avoiding misleading TPS estimates for very small initial decode segments.

• Operator status and access control

  • New private GET /routes endpoint provides a dashboard-oriented snapshot
    of public routes: resolved backend/model/capabilities, rolling activity and
    errors, performance averages, limits, and in-flight work.
  • Client API-key protection supports inherited api_keys at global or route
    scope using standard Authorization: Bearer <key> credentials. Client keys
    are never forwarded to the upstream backend.

• Safer request boundary and clearer metrics

  • New process-wide request_limits.max_body_bytes guard rejects oversized
    declared or chunked JSON requests before parsing (64 MiB default).
  • Performance accounting distinguishes logical prompt tokens from cached and
    uncached work, so prompt TPS is not inflated by KV-cache hits.
  • Each completed request emits execution.chat.performance_metrics in PLAIN
    logs with the dashboard's derived values.

• Qwen / LibreChat controls and configuration

  • Route overrides forward reasoning_effort as an OpenAI-compatible top-level
    request field.
  • The full configuration example documents transport, observability, status,
    API-key, and request-safety controls.

Reliability fixes

• Nudge continuation retries now apply the intended messages payload patch,
  preserving the lazy assistant prefix before the continuation prompt.
• Nudge, tool-call, reasoning, and terminal streaming paths have additional
  end-to-end regression coverage.

Validation

• 1,520 tests passed
• 27 intentional integration skips
• Clean public-repository validation completed successfully:
  • fresh Python/venv setup;
  • both configuration examples validated;
  • fake-backend quick start verified end-to-end;
  • independent clone of tag v0.9.3 verified without internal development
    directories.

Upgrade notes

• api_keys protects clients calling KRM. It is distinct from singular
  api_key, which authenticates KRM to an upstream provider.
• Treat GET /routes as an operational endpoint: expose it only on a trusted
  network or behind your own access control.
• Review config.example.full.yaml
  for the complete configuration surface, especially upstream_transport,
  observability, request_limits, and route inheritance.
• Existing routes continue to work without api_keys or request_limits;
  those controls are opt-in and have safe defaults.

See the README
and configuration guide
for setup and configuration details.