Skip to content

feat(routing): add opt-in knn task-level model routing backend#163

Merged
ypflll merged 12 commits into
mainfrom
feat/knn_model_router
Jul 21, 2026
Merged

feat(routing): add opt-in knn task-level model routing backend#163
ypflll merged 12 commits into
mainfrom
feat/knn_model_router

Conversation

@ypflll

@ypflll ypflll commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an optional task-level KNN model-routing backend alongside the existing EcoClaw router. At the start of each turn the router embeds the task, retrieves the nearest training tasks from a prebuilt memory, and picks the model with the best expected value (reward - lambda_cost * cost). Routing is opt-in and off by default (routing.enabled = false); when disabled the agent loop is unchanged.

Commit by commit:

  • config: RoutingConfig gains a backend switch (ecoclaw | knn) plus knn fields (models, memory_path, k, lambda_cost, embedding_endpoint) and safety-gate fields; all additive with defaults.
  • providers: PerModelProvider dispatches each call to a per-model endpoint by model name; unknown models fall back to the wrapped provider unchanged.
  • routing: KNNModelRouter exposes the same select_model_chain interface as the EcoClaw router; any failure yields (None, []) so the caller keeps its model.
  • cli: the gateway builds the router by backend and, for knn, wraps the provider in PerModelProvider.
  • routing (gates): leave the default model only with enough evidence - a memory-size floor, a count of similar neighbours (cosine >= min_similarity), and a margin over the default model's score; scoring uses only the similar neighbours. Defaults keep routing safe out of the box.
  • test: a real-LLM e2e drives AgentLoop._process_message over two OpenRouter models (self-skips without credentials).

Type

  • Fix
  • Feature
  • Docs
  • CI / tooling
  • Refactor
  • Other

Verification

Commands and results:

  • pytest tests/test_knn_router.py tests/test_config_routing.py tests/test_per_model_provider.py tests/test_cli_gateway_commands.py -> 70 passed
  • pytest tests/integration/test_knn_routing_real_llm.py -> 1 passed with credentials, skips cleanly without
  • ruff check on the changed files -> All checks passed
  • Full suite: 4047 passed; the remaining failures are pre-existing environment gaps (sandbox / TUI pty / Windows node runtime / channel SDKs) that reproduce on origin/main without this branch.

Offline routing reproduction against a prebuilt memory (the algorithm makes the live decision; the routed model's outcome is looked up from stored per-model trajectories):

  • ScienceWorld: KNN 58.47 vs Pure 27B 57.99 at ~13% lower cost.

  • AppWorld: KNN 55.3% pass vs Pure 27b 50.9%.

  • Relevant tests pass locally

  • Relevant lint / type checks pass locally

  • User-facing docs or screenshots are updated when needed

Risk

Off by default: with routing.enabled = false, build_model_routing returns the provider unchanged and the loop's routing block is skipped, so behaviour is identical to before. The new provider and router classes are only imported when the knn backend is enabled. Rollback is to disable routing in config or revert the branch.

  • Security impact considered
  • Backward compatibility considered
  • Rollback path is clear for risky changes

Related Issues

N/A

yaopengfei and others added 7 commits July 20, 2026 12:12
Extend RoutingConfig with a backend switch (ecoclaw | knn) and knn-specific fields (models, memory_path, k, lambda_cost, embedding_endpoint), plus a ModelEndpoint submodel. Additive with defaults; existing fields and the default ecoclaw backend keep current behavior.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
PerModelProvider dispatches each call to a per-model CustomProvider by model name; unknown models fall back to a wrapped provider unchanged. Used by the knn routing backend where routable models live on different OpenAI-compatible endpoints.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
KNNModelRouter embeds the incoming task, retrieves the k nearest training tasks from a prebuilt memory, and picks the model with the best expected value (reward - lambda_cost * cost) over those neighbours. Same select_model_chain interface as the ecoclaw ModelRouter; any failure returns (None, []) so the caller uses the default model.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
Extract build_model_routing() and pick the router by config.routing.backend: knn builds a KNNModelRouter and wraps the provider in PerModelProvider; ecoclaw keeps the PinchBench ModelRouter. Routing stays opt-in and off by default; the agent loop is unchanged.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
Leave the configured default model only with enough evidence, so an off-distribution query (e.g. casual chat) or a cold-start memory keeps the default instead of switching on a weak match. Score the pick over the similar neighbours (cosine >= min_similarity) rather than a fixed top-k. Gates: memory size, count of similar neighbours, and margin over the default model's score. Defaults keep routing safe out of the box (k=30, min_similarity=0.6, min_similar_neighbors=4, min_memory_size=10); KNNModelRouter takes the agent default model so the caller's model is the fallback.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
Drive AgentLoop._process_message with a real KNNModelRouter + PerModelProvider over two OpenRouter models: the simple task stays on the cheap default, the hard task switches to the strong model, and the routed model returns a real answer. Self-skips unless an embedding endpoint and OPENROUTER_API_KEY are available; sandbox/MCP bring-up is stubbed so a text turn runs without a VM.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
@ypflll ypflll changed the title Feat/knn model router feat(routing): add opt-in knn task-level model routing backend Jul 21, 2026
@ypflll ypflll closed this Jul 21, 2026
@ypflll ypflll reopened this Jul 21, 2026
@ypflll
ypflll requested a review from 0xKT July 21, 2026 02:59
@0xKT

0xKT commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

LGTM overall — CI is green and I ran the unit suite locally
(test_knn_router, test_config_routing, test_per_model_provider,
test_cli_gateway_commands), all pass.

One non-blocking nit before merge: raven/routing/knn_router.py imports numpy
at module top level, but numpy isn't a declared direct dependency — it's only
transitively locked (via everos -> lancedb), and this is the first shipped
raven/ source file to import it directly. Low risk since that chain is all-hard,
but it's inconsistent with the repo's own convention: pyproject.toml already
pins tiktoken directly for exactly this case ("... so the everos extractor keeps
working if upstream ever drops it"). Suggest uv add numpy with a similar
one-line comment.

yaopengfei and others added 5 commits July 21, 2026 06:53
Memory entries can store 'text' instead of a precomputed 'embedding'; the router embeds each at load via the configured endpoint (batch=1, cached under ~/.raven), so the file stays small and its vectors stay consistent with live queries. Precomputed embeddings are still honoured. Also score reward and cost over the same neighbours, and wrap the KNN path so any routing error degrades to the default model instead of failing the turn. Ships a small example memory (ScienceWorld + AppWorld).

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
raven/routing/knn_router.py imports numpy directly; it was only present transitively. Declare it so the dependency is explicit and the lockfile stays honest.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
Add the trailing newline end-of-file-fixer expects on the shipped example memory (was failing the pre-commit CI check), and note why numpy is a direct dependency, mirroring the tiktoken comment, per review.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
Bound the direct numpy dependency instead of leaving it unversioned, matching
the repo convention where every dependency carries a range. The lower bound
clears both lancedb (>=1.24.0) and everalgo-clustering (>=1.26); the <3.0 cap
surfaces a future numpy 3.x bump loudly rather than silently pulling an
untested major. Resolved numpy stays at 2.4.4.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
@ypflll
ypflll merged commit b913bec into main Jul 21, 2026
8 checks passed
@ypflll
ypflll deleted the feat/knn_model_router branch July 21, 2026 07:25
@0xKT 0xKT mentioned this pull request Jul 21, 2026
9 tasks
0xKT added a commit that referenced this pull request Jul 21, 2026
## Summary

Bump version from 0.1.6 to 0.1.7 for the next release. Touches
`pyproject.toml` and `uv.lock` only.

Release-scope changes since v0.1.6:
- feat(agent): offer deep-vs-regular choice for deep_research (#168)
- feat(routing): add opt-in knn task-level model routing backend (#163)
- fix(tracing): show model-call reasoning above the answer, not below
(#154)
- fix(tui): announce log path only on abnormal tui exit (#166)
- docs: update readme / chinese readme for recent features (#159, #160)

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [x] Other

## Verification

- [x] Relevant lint / type checks pass locally

```
uv lock                                                  # resolved, raven 0.1.6 -> 0.1.7
uv run python scripts/check_commit_messages.py origin/main..HEAD   # exit 0
```

## Risk

- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

Version-only bump; no code changes.

## Related Issues

N/A

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants