CacheRoute-v0.1.9

CacheRoute v0.1.9

CacheRoute v0.1.9 is a major prototype update focused on adaptive knowledge injection, queue-time prediction, topology-aware scheduling, resource observability, and performance experimentation.

Highlights

Adaptive Text/KVCache Injection

• Added the Intelligent Work Selection (IWS) strategy for dynamically choosing between text and kvcache injection.

• IWS compares estimated end-to-end costs, including:

  • ready-queue waiting time;
  • text prefill cost;
  • KVCache transfer time;
  • KDN link contention;
  • Redis-to-GPU loading cost;
  • remaining text recomputation cost.

• Added runtime and demo configuration through PROXY_INJECTION_STRATEGY and --injection-strategy.

• Improved decision logging and trace fields for analyzing the selected injection path and its predicted cost.

Queue Prediction and Scheduling

• Replaced the previous single-tail predictor with a dual-layer reservation model:

  • per-worker slot availability;
  • a shared per-Instance Prefill timeline.

• Added ordered prepare-to-ready sequencing with per-task sequence numbers and a prepared-task buffer.

• Added optional text_bypass ready-release policy so completed text tasks can bypass blocked KVCache tasks without stalling the release cursor.

• Added per-KDN-link KVCache transfer queues, serialization, prediction recomputation, and actual-time correction using kv_ack timestamps.

• Added cold-start penalties and online correction of pending task predictions.

• Refined prediction semantics around first-token completion and added decode-stage lifecycle and TPOT-related tracing.

• Expanded predicted and actual timing fields for queue wait, Prefill, KV transfer, Redis loading, first token, forwarding, and total latency.

CacheRoute Scheduler Strategy and Topology Awareness

• Added the cacheroute Scheduler strategy for knowledge- and topology-aware KDN/Proxy selection.

• KDN selection can now consider:

  • text and KVCache knowledge coverage;
  • KDN overload state;
  • pending and active network transfers;
  • network queue latency.

• Proxy selection can consider:

  • Instance-to-KDN topology;
  • Proxy load;
  • knowledge affinity;
  • configured safety windows.

• Added automatic Instance-to-KDN topology discovery using measured latency and interface bandwidth.

• Proxy aggregates Instance-reported links and reports the best available KDN links to the Scheduler.

• Added strategy and topology debug information for inspecting scheduling decisions.

KDN Network and KVCache Transfer Modeling

• Added real-link serialization for KDN text and KVCache transfers.
• Text retrieval is prioritized over queued bulk KVCache transfers to reduce user-facing blocking.
• Added transfer timestamps, queue delays, observed throughput, and bandwidth-utilization metrics.
• Added configurable Redis target rewriting to support cross-host and non-loopback experiments.
• Added KVCache size alignment and transfer-time prediction for Proxy scheduling.
• Corrected KDN-link timelines using actual injection completion times and recomputed predictions for waiting tasks.

TTFT, TPOT, and Redis Prediction Tooling

• Added Proxy-side TTFT regression and runtime prediction utilities.

• Added the instance/TPOT_predictor package for collecting per-token decode measurements across prompt-length and batch-size combinations.

• Improved TTFT warmup with:

  • per-configuration warmup;
  • non-blocking server startup;
  • configurable sample-selection policies;
  • unique prompt prefixes and noise;
  • minimum target-token enforcement;
  • per-request TTFT reporting.

• Added Redis pull-time regression and KVCache latency breakdown helpers.

• Improved knowledge-length calculation by preserving full KDN content and applying the target model tokenizer when possible.

Performance and Trace Collection

• Added client/kv_timing_sender.py for RPS-driven KVCache timing experiments and JSONL/CSV export.

• Expanded client/perf_client.py with:

  • configurable hybrid Text/KVCache patterns;
  • RPS and concurrent workload modes;
  • optional GPU utilization monitoring;
  • richer latency decomposition;
  • per-request trace printing;
  • trace-order and trace-integrity validation.

• Improved streaming metadata handling so cacheroute_meta is emitted more reliably before stream completion.

• Added additional Proxy and QueueManager timestamps covering routing, prepare, injection, ready dispatch, forwarding, first token, and failures.

Instance Resource Monitoring

• Added a lightweight Rust-based Instance Resource Agent.

• The agent collects:

  • CPU and system load;
  • memory usage;
  • network counters and link speed;
  • GPU utilization and memory state when available;
  • a basic admission-state hint.

• Added Resource Agent health and snapshot APIs:

  • GET /healthz
  • GET /v1/resource/snapshot

• Added a reporter that periodically sends Instance resource snapshots to the Proxy.

• Integrated Resource Agent startup, reuse, readiness checks, reporting, and cleanup into test/demo_instance.py.

• Added browser and Tkinter dashboards for local resource inspection.

• Added an end-to-end resource-monitoring smoke workflow.

Proxy and Scheduler Resource Observability

• Proxy now stores normalized per-Instance resource snapshots and freshness metadata.

• Added Proxy inspection APIs, including:

  • GET /debug/instance_resources
  • GET /debug/pool_resource
  • GET /v1/instance/list?include_dead=true

• Added compact Proxy-level pool_resource aggregation for:

  • Instance liveness and freshness;
  • inflight load and QPS;
  • CPU, memory, and GPU utilization;
  • GPU memory pressure;
  • pool admission state.

• Proxy reports pool_resource during Scheduler registration and heartbeat.

• Scheduler stores and exposes the reported Proxy-pool state for observability and future resource-aware strategies.

Browser-Based Proxy UI

• Added a browser-based Proxy observability dashboard under UI/proxy_ui/.

• The dashboard displays:

  • Proxy health;
  • alive, stale, and total Instances;
  • Instance resource snapshots;
  • topology and KDN-link state;
  • Scheduler registration state;
  • runtime summary cards and charts.

• Integrated UI startup, readiness probing, configuration, and cleanup into test/demo_proxy.py.

• Added a compact Proxy GET /debug/status API for UI and CLI consumers.

Documentation and Project Maintenance

• Added an Apache 2.0 license file.
• Expanded the root and component READMEs with architecture, startup, configuration, API, predictor, dashboard, and troubleshooting information.
• Added a frontend URL index for the available CacheRoute browser interfaces.
• Standardized comments, docstrings, logs, CLI messages, and documentation in English across the main project modules.
• Added a Rust-enabled development container configuration for Resource Agent development.
• Removed obsolete binary design documents in favor of repository-maintained Markdown documentation.

Compatibility and Operational Notes

• Existing OpenAI-compatible service-plane request formats remain supported.
• New injection strategies, release policies, monitoring tools, and resource reports are configurable and can be disabled when not required.
• The Rust toolchain is only required when building or running the native Resource Agent.
• Resource information reported to the Scheduler is currently used primarily for observability and future strategy development; it does not automatically replace existing routing policies.
• Validation across the development cycle included Python compilation checks, Rust cargo check, targeted smoke tests, end-to-end demo checks, and selected pytest runs. Not every incremental change was covered by a complete automated integration suite.