Add CacheRoute strategy, KDN knowledge index and pass request context to strategy.select

[zhy1658858023]

Motivation

• Provide a scheduler-side lightweight per-KDN knowledge metadata index so strategies can reason about which KDN can satisfy knowledge/RAG requests without re-fetching embeddings.
• Enable richer routing decisions by passing a normalized request_ctx into proxy/kdn selection strategies.
• Implement a first-phase CacheRoute strategy to pick KDNs that best cover requested knowledge and choose less-loaded proxies accordingly.

Description

• Added refresh_kdn_knowledge_index in scheduler/knowledge/kdn_sync.py to fetch per-KDN metadata (kid -> length/kv_ready) and store it on app.state.kdn_knowledge_index, and integrated its usage into kdn_refresh_once.
• Extended the strategy API in scheduler/strategy/base.py to accept an optional request_ctx and updated round_robin to keep the unified signature.
• Implemented CacheRouteStrategy in scheduler/strategy/cacheroute.py that ranks KDNs by text coverage / KV cache coverage / overload status and selects proxies by inflight/qps/gpu_util with stable tie-breaking.
• Registered the new strategy in scheduler/strategy/factory.py and added a --cacheroute shortcut in test/demo_scheduler.py.
• Enriched scheduler internals in scheduler/scheduler.py to prepare and pass request_ctx (including kdn_knowledge_index, knowledge_list, and other lightweight fields) and to send richer proxies/kdns metadata into strategy selection; adapted core/request.py to include request_ctx when calling strategy.select.

Testing

• Performed smoke import and instantiation checks by importing scheduler, CacheRouteStrategy, and calling select with simple/dummy inputs, which succeeded.
• Launched the demo startup path (demo_scheduler) with the --cacheroute shortcut to verify environment wiring and that the scheduler initializes without immediate exceptions.
• No new automated unit tests were added in this change.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fcb455e9e6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Isolate failing KDN snapshots during index refresh

This loop makes kdn_refresh_once depend on successful /knowledge/snapshot calls to every alive KDN: if one node times out or returns non-2xx, fetch_kdn_snapshot raises and the whole refresh aborts before the normal table-update path runs. In mixed-health clusters, one flaky KDN can block scheduler knowledge refresh globally and keep knowledge_table stale even when other KDNs are healthy.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Stop injecting full KDN index into request payload

Adding _scheduler_kdn_knowledge_index to the per-request payload makes each request carry the full scheduler-side knowledge index object, and _handle_client logs payloads when SCHEDULER_VERBOSE_REQUEST_LOG is enabled (default is 1 in core/config.py). With non-trivial knowledge sets this can massively inflate per-request log volume and request-time overhead, turning routing metadata into a hot-path performance issue.

Useful? React with 👍 / 👎.