Add CacheRoute strategy and KDN knowledge indexing; propagate request_ctx to scheduler and strategies

[zhy1658858023]

Motivation

• Introduce a KDN-aware scheduling strategy (CacheRoute) that can decide KDN and proxy selection using per-KDN knowledge metadata to improve routing decisions.
• Provide scheduler-side support to gather lightweight KDN knowledge metadata and pass it into selection strategies to avoid duplicated work (e.g. redoing embeddings or lookups in strategies).
• Make strategy interface and scheduler plumbing more extensible so future strategies can consume richer request context without changing call sites widely.

Description

• Added scheduler/strategy/cacheroute.py implementing CacheRouteStrategy which selects KDNs by text coverage / KV coverage / overload heuristics and selects proxies by low inflight / qps / gpu_util, and exposes get_debug_snapshot for /debug/strategy.
• Extended scheduler/strategy/base.py to accept a request_ctx optional parameter in the ProxySelectionStrategy.select signature to pass precomputed context into strategies.
• Updated scheduler/strategy/round_robin.py to accept the new request_ctx argument and retain compatibility.
• Registered the new strategy in scheduler/strategy/factory.py under the name cacheroute.
• Added refresh_kdn_knowledge_index to scheduler/knowledge/kdn_sync.py to build a lightweight per-KDN knowledge metadata index (mapping KDN -> {kid -> {length, kv_ready}}) and integrated its use into the KDN refresh flow so the scheduler keeps app.state.kdn_knowledge_index updated.
• Modified core/request.py to accept an extra kdn_knowledge_index argument in Request.build_request and to pass a compact request_ctx when calling strategy select (includes knowledge_list, knowledge_length, prompt_token_length, kdn_knowledge_index, etc.).
• Updated scheduler/scheduler.py to: provide richer proxy/KDN metadata to strategies, call _handle_client with kdn_knowledge_index, populate kdn_knowledge_index in app state, and include strategy debug snapshot in /debug/strategy output if available.
• Added small usability change in test/demo_scheduler.py to add a --cacheroute shortcut flag that sets --strategy cacheroute when launching the demo scheduler.
• Miscellaneous: preserved/cleaned up logging and debug outputs, and ensured the new fields are propagated through request creation and scheduling decisions.

Testing

• Performed smoke import checks for the modified modules with a simple import (import scheduler, import scheduler.strategy.cacheroute) and validated no immediate import errors (success).
• No automated test suite was executed as part of this change; manual demo script updated for local run with --cacheroute to exercise the new strategy during integration testing.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4f64e089b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Skip unreachable KDNs while building knowledge index

refresh_kdn_knowledge_index calls fetch_kdn_snapshot for every alive KDN without handling per-node failures. Since fetch_kdn_snapshot raises on timeout/HTTP errors and kdn_refresh_once invokes this step before any fallback selection, one temporarily unreachable KDN can abort the whole refresh pass, leaving both knowledge_table and kdn_knowledge_index stale even when other KDNs are healthy. This should degrade gracefully by catching errors per KDN and continuing with remaining nodes.

Useful? React with 👍 / 👎.