Repository navigation
semantic-router-counterfactual v0.1 #3946
ZC502
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Offline counterfactual auditing for routing decisions
What is this?
semantic-router-counterfactualis a small standalone prototype for auditing the stability of routing decisions under nearby counterfactual states.It is designed around Router Replay-style evidence, using a compact Counterfactual Audit Bundle that combines:
No changes to the production routing hot path are required.
A native vLLM Semantic Router Replay adapter would be a natural next step.
What does it measure?
The prototype currently answers three questions.
1. Counterfactual Safety Margin
What is the smallest signal change that causes the route to flip?
2. Route Divergence
When one signal changes, how much does the route fingerprint change across:
3. Order-Sensitive Residual
For genuinely stateful interventions, does:
produce a different final route than:
The tool does not manufacture order sensitivity by merely permuting independent static assignments. The bundled example uses explicit state transitions and a latched safety route.
Example
Fragile authorization boundary
Stateful order-sensitive transition
Design principles
eval()/exec()The bundled CI/CD gray-authorization fixture is synthetic and exists only to validate and demonstrate the auditor. It is not evidence of a Semantic Router implementation defect.
Resources
Source / README / synthetic fixture: https://gist.github.com/ZC502/c58135d0b093fc2a2a96820c6370cd33
Background
This prototype grew out of a broader residual-auditing line of work inspired by NARH — the Non-Associative Residual Hypothesis.
The recurring idea is not to judge a system from one successful execution, but to inspect how nearby alternative execution paths diverge.
Related applications I’ve explored include:
[MRV2 discussion slack ]
vLLM Slack, workspace access required, [analyze_mrv2_draft_parity.py-MRV2 Draft-Model Behavioral Parity Probe ][[Open Source Robotics Foundation/ OBIO discourse]
The abstraction is roughly:
This prototype explores the third category: decision-path residuals.
Feedback
I’d be interested in feedback from Semantic Router users and maintainers:
All reactions