v0.9.3 - Performance and Runtime Stabilization

KeepRoLLMing v0.9.3

KeepRoLLMing 0.9.3 is a performance and runtime-stabilization release. It
focuses on making long-lived, concurrent streams more dependable while giving
operators clearer visibility and safer controls around the proxy boundary.

Highlights

• More dependable long-running streams

  • A bounded shared upstream HTTP transport makes connection-pool behaviour
    explicit and observable.
  • Downstream cancellation now promptly closes the upstream stream, preventing
    backend generations from continuing after the client has gone away.
  • Graceful shutdown cancels background config watching and closes the shared
    HTTP client before observability workers stop.

• Observability that stays out of the request path

  • PLAIN, JSON, compact-server, and performance projections use independent
    bounded workers, so slow filesystem output cannot stall a response stream.
  • Lifecycle diagnostics cover downstream delivery failures, cancellation, and
    upstream request transitions.
  • Streaming progress now accounts for reasoning and tool-call output, while
    avoiding misleading TPS estimates for very small initial decode segments.

• Operator status and access control

  • New private GET /routes endpoint provides a dashboard-oriented snapshot
    of public routes: resolved backend/model/capabilities, rolling activity and
    errors, performance averages, limits, and in-flight work.
  • Client API-key protection supports inherited api_keys at global or route
    scope using standard Authorization: Bearer <key> credentials. Client keys
    are never forwarded to the upstream backend.

• Safer request boundary and clearer metrics

  • New process-wide request_limits.max_body_bytes guard rejects oversized
    declared or chunked JSON requests before parsing (64 MiB default).
  • Performance accounting distinguishes logical prompt tokens from cached and
    uncached work, so prompt TPS is not inflated by KV-cache hits.
  • Each completed request emits execution.chat.performance_metrics in PLAIN
    logs with the dashboard's derived values.

• Qwen / LibreChat controls and configuration

  • Route overrides forward reasoning_effort as an OpenAI-compatible top-level
    request field.
  • The full configuration example documents transport, observability, status,
    API-key, and request-safety controls.

Reliability fixes

• Nudge continuation retries now apply the intended messages payload patch,
  preserving the lazy assistant prefix before the continuation prompt.
• Nudge, tool-call, reasoning, and terminal streaming paths have additional
  end-to-end regression coverage.

Validation

• 1,520 tests passed
• 27 intentional integration skips
• Clean public-repository validation completed successfully:
  • fresh Python/venv setup;
  • both configuration examples validated;
  • fake-backend quick start verified end-to-end;
  • independent clone of tag v0.9.3 verified without internal development
    directories.

Upgrade notes

• api_keys protects clients calling KRM. It is distinct from singular
  api_key, which authenticates KRM to an upstream provider.
• Treat GET /routes as an operational endpoint: expose it only on a trusted
  network or behind your own access control.
• Review config.example.full.yaml
  for the complete configuration surface, especially upstream_transport,
  observability, request_limits, and route inheritance.
• Existing routes continue to work without api_keys or request_limits;
  those controls are opt-in and have safe defaults.

See the README
and configuration guide
for setup and configuration details.

v0.9.2 - Consolidation and Refactor, Reasoning support and External Prompts

KeepRoLLMing v0.9.2

KeepRoLLMing 0.9.2 is a major internal consolidation release: the proxy now has a canonical streaming path, modular filters, structured observability, stronger end-to-end coverage, and a cleaner public onboarding experience.

Highlights

• Canonical streaming pipeline

  • One authoritative streaming implementation for SSE parsing, accounting, finalization, serialization, and terminal framing.
  • Preserves OpenAI-compatible ordering for reasoning, tool calls, finish_reason, and [DONE].
  • Improved cancellation and keepalive handling for long-running streams.

• Modular filter architecture

  • Filters now live under keeprollming.filters, with each module owning its request/stream behavior and configuration validation.
  • Route configuration uses the simpler canonical filters: format.
  • Built-in modules include system prompts, summarization, multimodal validation, tool rewriting, timestamps, model nudge, tool-loop stopping, and reasoning-loop stopping.

• Observability and debugging

  • Event-based PLAIN and JSON logging with clearer transcript rendering.
  • Per-request performance events and cache-aware throughput metrics.
  • Opt-in raw SSE and request-body capture for focused production debugging.

• More reliable tool and reasoning flows

  • Correct handling of streamed tool-call fragments and terminal tool_calls finish reasons.
  • Nudge continuations preserve the original assistant prefix and retry with the intended request payload.
  • Tool/reasoning loop safeguards are covered by dedicated regression tests.

• Configuration and routing

  • New config.example.full.yaml documents the complete supported configuration surface.
  • System prompts can be loaded from files.
  • Route-level reasoning_effort overrides support LibreChat/Qwen-style controls.
  • Public runtime prompt templates are now shipped under prompts/.

• Onboarding and developer experience

  • Canonical local launcher: ./krm serve, ./krm start, ./krm status.
  • Verified Python/venv setup flow and deterministic fake-backend quick start.
  • Updated README, configuration, deployment, troubleshooting, and contributor documentation.

Validation

• 1480 tests passed
• 27 intentional integration skips
• Clean clone validation completed successfully:
  • fresh virtualenv setup
  • both configuration examples validated
  • fake-backend quick start verified end-to-end

Upgrade notes

This release intentionally removes legacy internal paths and obsolete launcher scripts. Use:

bash scripts/setup.sh
./krm serve --port 8000 --config config.yaml

For configuration, prefer the new canonical filters: route section and use
[config.example.full.yaml](https://github.com/danielebruneo/KeepRoLLMing/blob/v0.9.2/config.example.full.yaml)
as the complete reference.

See the README and configuration guide
for setup and migration details.

v0.9.1 - Remote API support

• Filter pipeline stability fixes
• Remote API support