Repository navigation
Model Intelligence
Summary of the
model_intelligencesystem: sourcing, schedule, storage, and consumption. Codebase:v3.8.52· Last verified: 2026-10-22
"AA" = Arena AI (LMArena-style human-preference ELO), not "Artificial Intelligence".
It is a crowd-sourced quality signal. Humans compare two model outputs, pick the better one, and the result updates an ELO rating per model. OmniRoute fetches those ELO ratings, normalizes them to a task-fit score in [0.4, 0.98], and uses them to rank providers and to bias routing.
| Source | Upstream | Module |
|---|---|---|
arena_elo |
https://api.wulong.dev/arena-ai-leaderboards/v1/leaderboard |
src/lib/arenaEloSync.ts |
models_dev_tier |
https://models.dev/api.json (via capabilities sync) |
src/lib/db/modelIntelligence.ts |
user_override |
Operator API (POST /api/intelligence/overrides) |
src/lib/db/modelIntelligence.ts |
| Setting | Default | Env var |
|---|---|---|
| Interval |
86400 s (24 h) |
ARENA_ELO_SYNC_INTERVAL |
| Enabled | true |
ARENA_ELO_SYNC_ENABLED (also dashboard feature flag) |
| First sync | Immediately at server boot (src/instrumentation-node.ts → initArenaEloSync()) |
non-blocking, never fatal |
| Freshness guard | If last sync is younger than interval, skip fetch | getLatestSyncedAt("arena_elo") |
| Failure backoff | After failure, skip for interval/4 (6 h default) |
key_value[arena_elo/lastFailedAt] |
| Manual trigger |
POST /api/intelligence/sync (management auth) |
dryRun: true previews without writing (Arena by default, the tier rebuild when combined with rebuildTier); rebuildTier: true rebuilds only the tier source; syncArenaElo: true forces the Arena sync alongside it. An Arena failure answers 502; a body with no source flag is an Arena sync as before |
| Status + health | GET /api/intelligence/sync |
returns enabled, lastSync, nextSync, intervalMs, sources, plus sourcesDetail with per-source count/lastSync/oldestExpiresAt
|
| Clear | DELETE /api/intelligence/sync |
deletes all arena_elo rows |
Failure never crashes the server. Rankings simply show last good data or empty state.
models_dev_tier rows are rebuilt automatically at the end of every successful capabilities save:
-
saveModelsDevCapabilities()insrc/lib/modelsDevSync.ts(full replace) -
upsertSyncedCapabilities()insrc/lib/modelsDevSync.ts(incremental upsert)
Errors are swallowed with console.warn so the capabilities save is never rolled back.
Manual rebuild: POST /api/intelligence/sync { "rebuildTier": true }.
taskFit = 0.4 + 0.58 * ((elo - minElo) / (maxElo - minElo || 1))
- Computed per leaderboard (min/max of that leaderboard).
- Range
[0.4, 0.98]— never 0 or 1, leaves headroom for user overrides. - Rounded to 4 decimals.
Category mapping:
| Arena leaderboard | OmniRoute task categories |
|---|---|
text |
default, review, documentation, debugging
|
code |
coding |
Other steps:
- Strip vendor prefixes (
anthropic/,openai/,google/, ... 18 total) and harness annotations ((codex-harness)). - Lowercase.
- Confidence from votes:
high>=5000,medium>=1000,low<1000. - Synthesize base rows for effort-suffix variants via
resolveScoresAs()(best variant score becomes base score).
TTL: expires_at = now + 7 days. Reader filters expires_at IS NULL OR expires_at > now().
buildModelsDevTierEntries() reads aggregated model_capabilities (capability-maximal booleans + max limit_context across providers for the same model id), derives tier via deriveTierFromCapabilities(), then maps tier → score via TIER_TASK_FITNESS:
| Tier | Confidence | Scores (coding/review/planning/analysis/debugging/documentation/default) |
|---|---|---|
premium |
high |
0.92/0.93/0.94/0.95/0.90/0.88/0.85 |
standard |
medium |
0.85/0.84/0.85/0.85/0.82/0.85/0.78 |
fast |
low |
0.78/0.72/0.70/0.72/0.75/0.80/0.72 |
budget |
low |
0.65/0.60/0.55/0.58/0.60/0.70/0.55 |
The table lives in open-sse/services/autoCombo/tierFitness.ts and is shared by the live resolver (taskFitness.ts) and the persisted rebuild, so the two cannot drift.
eloRaw = null, expiresAt = null (never expires — rebuilt on next capabilities sync).
Lifecycle veto: ids the vendor has retired (isVendorRetiredId(), backed by config/quality/model-lifecycle.json) are excluded.
user_override entries are written via POST /api/intelligence/overrides:
{ "model": "gpt-4o", "category": "coding", "score": 0.95 }-
scoreclamped to[0, 1]. -
categorymust be one of:coding | review | planning | analysis | debugging | documentation | default. -
expiresAt = null(never expires until explicitly deleted). -
DELETE /api/intelligence/overrides?model=gpt-4o&category=codingremoves the entry idempotently. - Both routes require management authentication.
Migration 097_model_intelligence.sql:
CREATE TABLE model_intelligence (
model TEXT NOT NULL,
source TEXT NOT NULL, -- arena_elo | models_dev_tier | user_override
category TEXT NOT NULL, -- coding | review | planning | analysis | debugging | documentation | default
score REAL NOT NULL, -- [0..1]
elo_raw INTEGER, -- original ELO if arena_elo
confidence TEXT, -- high | medium | low
synced_at TEXT NOT NULL DEFAULT (datetime('now')),
expires_at TEXT, -- NULL = never expires
PRIMARY KEY (model, source, category)
) WITHOUT ROWID;Indexes: (model, category), (source), (expires_at) WHERE NOT NULL.
Domain module: src/lib/db/modelIntelligence.ts.
| Function | Role | Production caller |
|---|---|---|
applyArenaEloRefresh(entries) |
Atomic upsert+prune in one tx | arenaEloSync.ts |
buildModelsDevTierEntries(caps?) |
Derive tier rows from capabilities | rebuildModelsDevTierIntelligence() |
applyModelsDevTierRefresh(entries) |
Atomic upsert+prune for tier source | rebuildModelsDevTierIntelligence() |
rebuildModelsDevTierIntelligence() |
Full tier rebuild (read caps → derive → persist) |
modelsDevSync.ts, POST /api/intelligence/sync
|
getIntelligenceSourcesHealth() |
Per-source count/lastSync/oldestExpiresAt | GET /api/intelligence/sync |
setUserFitnessOverrideEntry |
Write user_override | POST /api/intelligence/overrides |
deleteUserFitnessOverrideEntry |
Delete user_override | DELETE /api/intelligence/overrides |
listModelIntelligence({source, category}) |
List/filter | freeProviderRankings.ts |
getModelIntelligence(model, category) |
Priority-ordered lookup (override > elo > tier) | none in production code at this revision (tests only) |
getModelIntelligenceBySource |
Single-source lookup | taskFitness.ts::queryModelIntelligence |
getLatestSyncedAt(source) |
Freshness guard |
arenaEloSync.ts, rebuildModelsDevTierIntelligence()
|
| Source | Written by | Persisted rows | Notes |
|---|---|---|---|
arena_elo |
24 h periodic Arena sync | Yes — TTL 7 days | ELO + confidence per model/category |
models_dev_tier |
Capabilities sync success hook | Yes — no TTL | Rebuilt atomically; prunes stale rows |
user_override |
POST /api/intelligence/overrides |
Yes — no TTL | Operator-set; persists until DELETEd |
src/lib/freeProviderRankings.ts::computeFreeProviderRankings()
- Iterates
getFreeProviders()= allNOAUTH_PROVIDERS+OAUTH_PROVIDERS/APIKEY_PROVIDERSwhereproviderHasFreeModels(id). - For each provider, loads
REGISTRY[provider].models+ custom models, thenfindMatchingIntelligence()per model:- Exact match on lowercased id
- Strip trailing version suffix (
kimi-k2.6->kimi-k2) - Prefix match (
intel modelis prefix ofregistry id)
- Keeps only models with a match; sorts models by score; provider ranked by
topModel.scorethenaverageScore. - Exposed at
GET /api/free-provider-rankings?category=&limit=and dashboard/dashboard/free-provider-rankings.
open-sse/services/autoCombo/taskFitness.ts::getTaskFitnessWithSource(model, taskType)
Resolution chain per request (highest priority first):
-
user_overrideDB hit -
arena_eloDB hit - Inherited retry of 1-2 via
resolveScoresAs()base id (:inherited) -
models_dev_tierDB hit — now populated on capabilities sync - Static
FITNESS_TABLE(small, versioned ids only) - Wildcard boosts (
coder,code,fast,thinking) over 0.5 baseline
Retired ids (via model-lifecycle.json) veto layers 1-4 and return neutral 0.5.
Consumed by scoring.ts::scorePool() (16-factor auto-combo scoring) and comboScoringInspector.
model_intelligence.model is a bare model name (claude-opus-4-6), not provider/model. The Arena sync strips vendor prefixes before storing. Matching is by model id, so the same score applies regardless of which provider serves that model.
-
UPSTREAM_PROXY_PROVIDERS(cliproxyapi,9router) are not iterated bygetFreeProviders(). -
AGGREGATOR_PROVIDER_IDS(~50 ids) are inREGISTRYand eligible for rankings and routing. Their registry model ids are matched against bare Arena/tier names viafindMatchingIntelligence/resolveScoresAs.
GET /api/intelligence/sync now returns an additional sourcesDetail field:
{
"enabled": true,
"lastSync": "2026-10-07T12:00:00Z",
"nextSync": "2026-10-08T12:00:00Z",
"intervalMs": 86400000,
"sources": ["arena_elo"],
"sourcesDetail": {
"arena_elo": {
"count": 1842,
"lastSync": "2026-10-07T12:00:00Z",
"oldestExpiresAt": "2026-10-14T12:00:00Z"
},
"models_dev_tier": {
"count": 4312,
"lastSync": "2026-10-07T11:00:00Z",
"oldestExpiresAt": null
},
"user_override": { "count": 3, "lastSync": "2026-10-07T10:00:00Z", "oldestExpiresAt": null }
}
}-
src/lib/arenaEloSync.ts— fetch, transform, schedule, status -
src/lib/db/modelIntelligence.ts— table + CRUD + tier rebuild + health -
src/lib/db/migrations/097_model_intelligence.sql— schema -
src/lib/freeProviderRankings.ts— rankings join + matching -
src/lib/modelsDevSync.ts— models.dev capabilities (tier rebuild wiring) -
open-sse/services/autoCombo/taskFitness.ts— routing fitness chain -
open-sse/services/autoCombo/tierFitness.ts— shared tier → fitness table -
open-sse/services/autoCombo/scoresAs.ts— variant->base resolution -
src/app/api/intelligence/sync/route.ts— manual sync/status/clear + rebuildTier + sourcesDetail -
src/app/api/intelligence/overrides/route.ts— POST/DELETE user_override entries -
src/app/api/free-provider-rankings/route.ts— rankings API -
docs/guides/FREE_PROVIDER_RANKINGS.md— user-facing guide
OmniRoute · Website · npm · Docker Hub
- Setup Guide
- User Guide
- Features
- Quick Start (Docker)
- Electron Desktop App
- Termux (Android)
- PWA Guide
- MCP Server
- A2A Server
- Agent Protocols
- OpenCode Plugin
- Webhooks
- Cloud Agents
- Skills
- Memory
- Evals
- Gamification
- Guardrails
- Compliance
- Error Sanitization
- Public Credentials
- Route Guard Tiers
- Stealth Guide
- CLI Token Auth