Repository navigation
Replies: 1 comment
|
I like the plan-binding idea, especially the expiry and audit-history treatment. It turns routing overrides into managed operational artifacts rather than hidden config drift. On the discussion points:
The key control is plan binding as an auditable release artifact: owner, rationale, expiry, affected traffic, safety plugins preserved, rollback path, and post-expiry review. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
The Problem
vLLM Semantic Router's routing pipeline is fully automatic:
This works for the happy path. But there are moments when operators or users need to say "this time, route to model X." Today, you can't:
curl -H "hint: model=gemini"for a testSemantic Cache solves latency, not correctness. And wrong cached routing decisions amplify the problem.
The Proposal
Two complementary mechanisms, inspired by 25+ years of database optimizer evolution (Oracle
/*+ INDEX */hints, PostgreSQLpg_hint_plan, TiDB SQL Plan Management):1. Routing Hints — request-level, ephemeral overrides
model=Xdecision=Xexclude_models=A,Ballow_models=A,Bsignal=type/nameThree injection sources, checked in priority order:
2. Plan Bindings — persistent, registry-level rules with lifecycle management
Bindings auto-disable on expiry — they don't auto-delete, preserving audit history and enabling rollback.
Key Design Decisions
Safety is non-negotiable:
authz,jailbreak,pii,kb,modalitycannot be injected as signalsx-vsr-enable-route-hints: true)Hints don't bypass safety plugins:
jailbreakandpiidetection always run (~5-8ms for in-process BERT classifiers)model=Xskips the full 17-classifier signal evaluationHints don't pollute learning loops:
hint_applied: true,feedback_eligible: falseEscape hatches are temporary:
Why Now
We surveyed 9 commercial platforms + 14 academic papers (2025-2026). No competitor combines all three layers:
/*+ VSR: ... */) — unique. Kong and Portkey support header-based overrides, but no one puts hints in the prompt itself.Database optimizers evolved from "purely automated" to "automated + hints + plan baselines" over 25+ years. LLM routing is still in phase 1. This is the right time.
Discussion Points
model=Xhints use the resolved Decision's plugin chain (system_prompt, header_mutation, RAG)?model=Xhinted responses ever update the semantic cache?/*+ VSR: model=X */the right annotation syntax?All reactions