Custom UI and combo policy for 36 auto channels (auto/best-coding, auto/claude-opus, etc.)

[manchairwang]

Issue: Now all these 36 auto channels (auto/best-coding, etc.) and 2 claude auto channels look like black box and are hard to contorl.

Propose:

E.g. For auto/best-coding channel, have a UI for this channel to

• remove some candidates which never work
• understand which candidate are online for this auto channel
• select sequence for all candidates at a time point,
• give some channel some + weight or -weight
• select a policy (e.g. sequence, auto, p2c, by remained quota, etc.)

auto/claude-opus and auto/claude-sonnet need this function as well.

[diegosouzapw]

@manchairwang -- agreed on the diagnosis: the auto/* channels are opaque in exactly the way you describe. You can pick one, but not see what it holds or why it chose what it chose, and a candidate that is permanently broken for your account stays in rotation with no way to remove it.

Opened #7819 to track the control surface.

Worth knowing that two of the five things you asked for are mostly a surfacing problem rather than new engineering:

• Policy selection -- open-sse/services/combo.ts already implements 18 strategies (priority, weighted, round-robin, p2c, cost-optimized, headroom, and more). Letting you choose one per auto channel is largely exposing what is already there.
• "Which candidates are online" -- OmniRoute already tracks this across three resilience layers (provider circuit breaker, per-connection cooldown, per-model lockout). The same state the health endpoint reads can drive that display.

The genuinely new parts are the per-user overrides: excluded candidates, per-candidate weights, and explicit ordering. Those need persistence of their own, since they are your preferences layered on top of the shared scoring rather than a property of the channel.

If you want the control today at the cost of the auto-scoring, building an explicit combo by hand gives you full candidate and strategy selection. Context on how the current scoring picks candidates is in docs/routing/AUTO-COMBO.md.

[manchairwang]

Yes I'm tring this. But it didn't work well on my side. Let me learn the doc carefully. I don't know it before, thx~

[diegosouzapw]

Makes sense, @manchairwang -- and no problem, the manual-combo route is not obvious until you have seen it once. The proper per-channel control surface you asked for is tracked in #7819; the hand-built combo is just the workaround until that lands.

When you say it "didn't work well" -- what happened? A couple of usual snags with hand-built combos:

• Candidates you want to exclude have to be left out of the combo entirely (there is no per-candidate mute yet -- that is part of feat(routing): per-candidate control + policy selection UI for auto/* channels #7819).
• The strategy you pick (priority / weighted / p2c / headroom / cost-optimized ...) changes ordering a lot; docs/routing/AUTO-COMBO.md has the full table and what each one optimizes for.

Tell me which strategy you tried and what it did vs. what you expected, and I will point you at the right one -- or confirm it is a gap that belongs in #7819.

[manchairwang]

I'm creating a new routing policy, whichi combine several light-weighted model together with a math methodology, to achieve top model benchmark. It's not your out-of-box strategy, it's a new one. But now it doesn't work well on multiple benchmarks. Only got top score on 1 benchmark.

For 18 out-of-box strategies, is there any detail document for them? I could not understand some of them, e.g. heardroom, reset-aware RR, Random / strict random, context relay, pipeline, etc.

[diegosouzapw]

Hey @manchairwang! For the full list of the 18 strategies with a one-line description each -- including headroom, reset-aware, strict-random, context-relay, and pipeline -- see docs/routing/AUTO-COMBO.md under "All Routing Strategies". There's also a deeper per-strategy breakdown further down ("Router strategies in detail") for the most commonly used ones.

On your custom weighted-combination policy only topping one benchmark: that's usually a sign the weights are overfit to whichever benchmark you tuned against first. Worth checking whether the same weights still win if you re-derive them per-benchmark instead of using one global set. Happy to take a look at the actual weights/formula if you want a second pair of eyes.

[manchairwang]

Thx! This is exactly what I'm looking for: docs/routing/AUTO-COMBO.md

[diegosouzapw]

Glad that's the one, @manchairwang! If you do get your weighted-combination policy converging across benchmarks, I'd be interested in seeing the numbers -- and the per-channel control surface you originally asked for is still tracked in #7819.

[manchairwang]

It's a complex process, not just weight. It's still in early stage, which means reorg daily...

[diegosouzapw]

Understood, @manchairwang -- if the whole thing is reorganizing daily then that is genuinely research rather than tuning, and topping one benchmark out of several is the expected shape at that stage. Three things from the code that may save you some time:

1. fusion is the closest built-in to what you are describing. It is the one strategy that does not pick a single target: it fans the prompt out to a panel of models in parallel and then has a judge model synthesize one final answer (open-sse/services/fusion.ts, documented in docs/routing/AUTO-COMBO.md under the strategy table and the "Fusion" section). If your methodology is "combine several light models mathematically to reach top-model quality", that is the same shape at a high level. Worth running as a baseline -- if your approach does not beat fusion on a given benchmark, that is a much cheaper signal than a full benchmark sweep.

2. Strategies are not pluggable today. They are resolved by literal if (strategy === "...") branches inside open-sse/services/combo.ts -- there is no registry or extension point to drop a module into. A genuinely new policy means patching that file, which is worth knowing before you design against an interface that does not exist. If you get yours working and want it upstream, that is the file it would live in.

3. For the cross-benchmark measurement problem itself, OmniRoute ships an eval framework (src/lib/evals/, docs in docs/frameworks/EVALS.md) that can run the same suite against different routing configurations. That is a more reliable way to tell "this change helped everywhere" from "this change helped the benchmark I tuned against" than comparing runs by hand -- which is the specific failure mode a single-benchmark win usually indicates.

The per-channel control surface you originally opened this for is still tracked in #7819.

[manchairwang]

Understood, @manchairwang -- if the whole thing is reorganizing daily then that is genuinely research rather than tuning, and topping one benchmark out of several is the expected shape at that stage. Three things from the code that may save you some time:

1. fusion is the closest built-in to what you are describing. It is the one strategy that does not pick a single target: it fans the prompt out to a panel of models in parallel and then has a judge model synthesize one final answer (open-sse/services/fusion.ts, documented in docs/routing/AUTO-COMBO.md under the strategy table and the "Fusion" section). If your methodology is "combine several light models mathematically to reach top-model quality", that is the same shape at a high level. Worth running as a baseline -- if your approach does not beat fusion on a given benchmark, that is a much cheaper signal than a full benchmark sweep.

Yes I'm create a compelx version of fusion.

2. Strategies are not pluggable today. They are resolved by literal if (strategy === "...") branches inside open-sse/services/combo.ts -- there is no registry or extension point to drop a module into. A genuinely new policy means patching that file, which is worth knowing before you design against an interface that does not exist. If you get yours working and want it upstream, that is the file it would live in.

Yes I'm writing my code here.

3. For the cross-benchmark measurement problem itself, OmniRoute ships an eval framework (src/lib/evals/, docs in docs/frameworks/EVALS.md) that can run the same suite against different routing configurations. That is a more reliable way to tell "this change helped everywhere" from "this change helped the benchmark I tuned against" than comparing runs by hand -- which is the specific failure mode a single-benchmark win usually indicates.

The per-channel control surface you originally opened this for is still tracked in #7819.
Thanks! I never found this eval framework can be used this way. Let me have a try.

[c111mb3r]

Hello OmniRoute Team,

I hope you are doing well.

I am following up on the OmniRoute ACP Custom-Agent Remote Code Execution vulnerability that I reported approximately three weeks ago. I understand that the issue has now been fixed, and I appreciate your team’s efforts in addressing it.

Since the vulnerability had remote code execution impact, I wanted to ask whether your team would be open to requesting a CVE identifier for it. I would also be interested in coordinating a responsible public disclosure once you consider it appropriate.

Please let me know your thoughts, as well as whether there is a preferred process or disclosure timeline that I should follow. I would be happy to provide any additional technical details needed for the CVE request.

Thank you again for your time and for addressing the issue.

[diegosouzapw]

Quick housekeeping on this thread: the CVE/disclosure question from @c111mb3r is being handled in #8850 (the dedicated security thread), so no need to track it here.

And for @manchairwang -- the per-channel control surface you originally opened this for (#7819) has been completed and shipped. The eval framework tip (src/lib/evals/, docs/frameworks/EVALS.md) is the right tool for cross-benchmark validation of your custom fusion variant. Good luck with the research!