free-best-router — OpenAI-compat routing layer for DSH across 7 free providers (52 tests, 7-component scoring, Wilson + Bayesian, bounded fallback)

[MajidAsghariTabrizi]

What it is

free-best-router is a local Node.js process (5 source files, ~80 KB, one dep — undici) that exposes one OpenAI-compatible endpoint (http://127.0.0.1:4098/v1, logical model free-best) and continuously picks the best currently-usable free model across 7 providers: OpenRouter, OpenCode/Zen, Groq Cloud, Cerebras Inference, Mistral La Plateforme, DeepSeek, and local Ollama / LM Studio / llama.cpp. DSH treats it as one provider block; the router handles everything else.

It is not a native DSH plugin. It is a routing layer that DSH can call as an OpenAI-completions provider.

Why I built it

The free-model layer under DSH has properties that a model registry doesn't capture:

• Catalogs change every 6 hours
• 429 windows span 60 s to 8 min
• 404 cycles where providers remove a model for hours at a time
• Inference latency variance of 10x across healthy models
• Provider-level incidents take every model from that provider out at once

If an agent calls a specific free model by name, it inherits all of that. The router sits above providers so the agent never has to know which free model is healthy at 3am.

How it works

DISCOVER -> NORMALIZE -> FREE FILTER -> CAPABILITY FILTER -> HEALTH FILTER
        -> QUALITY RANK -> TASK RANK -> SELECT

Hard requirements (must all pass): free, provider reachable, code_generation, tool_calling; vision when images are present; long_context for repo-wide tasks.

Total score (src/classify.mjs):

total = capabilityFit*0.25 + agenticQuality*0.20 + reliability*0.20
      + codingQuality*0.15 + contextFit*0.08 + toolQuality*0.07 + latency*0.05

reliability is multiplied by providerMultiplier (1.0 healthy, 0.5 degraded, 0 cooldown), finishCompat, and a confidence-adjusted raw rate:

• 0 requests -> 0.5 prior
• < minSamples (default 5) -> Bayesian shrinkage toward 0.5 prior
• >= minSamples -> Wilson lower bound (z=1.96)

So a 1/1 model can't outrank a 200/205.

Time-decay penalties (configurable halflives) subtract from total for recent 429 / timeout / 5xx / tool-fail / structure-fail / finish-error events.

DeepSeek Harness integration

This is the part DSH maintainers will care about. It is a config block, not a plugin:

# settings.yaml
llm-pi-ai:
  providers:
    free-router:
      displayName: Free Best Router
      api: openai-completions
      baseURL: http://127.0.0.1:4098/v1
      apiKeyEnv: FREE_ROUTER_API_KEY
      models:
        - id: free-best

DSH's llm-pi-ai openai-completions adapter hits 127.0.0.1:4098/v1/chat/completions. The router selects a concrete model, preserves the upstream response's model field (so DSH captures the actual picked model into responseModel), and exposes the same value via the x-free-router-model response header.

A complete worked example is at examples/deepseek-harness/settings.yaml.

Reliability / fallback behaviour

• On 429: 60 s cooldown base, exponential (60 -> 120 -> 240 -> 480 s, capped at 8 min) per type
• On 404 / model_not_found: 30-min base, capped at 1 h - lets the free-tier cycle in/out naturally instead of hammering
• On empty pool: forced catalog refresh + re-rank once; if still empty, returns 502 with Retry-After (from the soonest-to-recover cooldown, clamped to [1, 600] s) and x-free-router-next-eligible (the model that will be eligible first)
• On 404 specifically: also forces a per-provider catalog refresh
• Provider-level cooldown excludes every model from that provider (vs per-model cooldown)
• Anti-correlated-burst reset: when a model's cooldown expires without a fresh failure, the consecutive counter is halved so the next 429 doesn't push every model to the 8 min cap simultaneously
• Bounded wall time: maxDepth: 4 x attemptTimeoutMs: 45_000 = 180 s maximum per request

Smart exploration

5% of requests probe a healthy, capability-close runner-up (not the incumbent) so the winner doesn't ossify. The pool is restricted to: free-only, healthy-state, and either (a) within exploreGap: 0.15 of the incumbent score or (b) under-sampled (requests < minSamples) and capability-close. Exploration outcomes are tracked separately so we can verify the policy is doing what it claims.

What's verified and what's not

• 52 deterministic unit tests across 6 test files (engine, multi-provider, adaptive exploration, exhaustion recovery, rotation, server fail-fast). No network, no secrets, no LLM calls. The test runner is node --test and runs in CI via .github/workflows/ci.yml.
• Zero telemetry - grep of src/ and providers/ shows the only outbound traffic is to configured provider baseUrls. No analytics SDK, no metrics endpoint, no third-party call. engine.diagnostics() asserts that no sk-..., OPENROUTER_OX, OPENCODE_ZEN, or DEEPSEEK_API string ever appears in the diagnostics JSON.
• Free-only invariant is enforced in code, not just config: hardEligible(model, task) returns { ok: false, reason: "paid" } if model.free is false. Tested.
• State is currently in-memory only - cooldowns, success rates, and per-model stats die with the process. The durable fix (serialize to ~/.storages/free-router/state.json and restore on boot) is the next change. Calling it out so people who use this in production know.
• DSH's own llm-pi-ai Retry-After behaviour (see [Bug] dsh-llm-pi-ai ignores Retry-After when retrying — TPM-limited retries always fail #3128) means the router's 502/429 hints currently can't be read by DSH's retry loop without a small adapter change. Happy to send a PR for that.

Repository

https://github.com/MajidAsghariTabrizi/free-best-router

• README.md - full quick-start, API reference, DSH config
• docs/architecture.md - engine internals, scoring math, cooldown state machine
• docs/providers.md - per-provider notes (rate limits, free-tier caveats)
• examples/curl/, examples/openai-python/, examples/javascript/, examples/deepseek-harness/

Feedback requested

• Anyone running this alongside dsh-quota-router / dsh-shift-router / dsh-zen-proxy / dsh-polyglot / dsh-adaptive-model-router - would love to know whether they compose cleanly or compete.
• Anyone with strong opinions on the 1 h cap for model_not_found cooldown - is that too long in practice?
• Maintainers: would the free-router provider block be a useful worked example in the self-hosted-gateways proposal (Proposal: make self-hosted gateways first-class providers in deepseek-harness (split `packages/llm`) #2529)?