Design: Policy-driven multi-source quota router for DSH

[MajidAsghariTabrizi]

Summary

This proposal explores a reusable quota-aware routing layer for DeepSeek Harness that separates routing policy from provider-specific execution.

The goal is to make multi-source model routing predictable, extensible, and independently testable, while allowing applications to define different routing policies for different workloads.

This follows the direction discussed in Discussion #3544.

Motivation

Applications using multiple model sources may need to:

• route requests across multiple providers or quota sources
• prioritize sources differently depending on the task
• avoid unhealthy or temporarily unavailable sources
• fall back automatically when a preferred source cannot serve a request
• keep routing decisions independent from provider-specific implementation details

Today, these concerns can easily become coupled together, making routing behavior harder to reason about and harder to reuse.

The proposed design introduces a small policy layer that decides where a request should go, while provider adapters remain responsible for how the request is executed.

Proposed Architecture

Request flows through: Task Profile -> Routing Policy -> Health / Quota Evaluation -> Source Adapter -> Response.

1. Task Profiles

A task profile describes the routing requirements for a request. Examples: default, coding, interactive, background, high-priority. A profile should not contain provider-specific implementation details.

2. Source Priority

The policy determines the preferred ordering of available sources.

3. Health-Aware Routing

A source should only be selected when it is currently eligible. Eligibility may consider availability, recent failures, quota state, cooldown state, provider health, and temporary rate limiting.

4. Deterministic Fallback

When the preferred source cannot serve a request, the router should move through the configured priority order. Routing decisions should be deterministic for the same policy and source state.

Separation of Responsibilities

Policy layer: task profiles, source priority, fallback rules, eligibility decisions, deterministic routing decisions.

Provider/source layer: authentication, API requests, provider-specific request/response handling, provider-specific health checks, quota reporting.

Package Boundary

A key question from the discussion is whether the routing layer should be packaged independently through npm. The preferred direction is to expose a small public interface rather than coupling consumers to internal implementation details.

Initial Scope

In scope: policy model, task profiles, ordered source selection, health-aware eligibility, deterministic fallback, provider-independent interfaces, unit tests, minimal public package interface.

Out of scope for the initial PR: complex load balancing, automatic provider discovery, billing/account management, distributed quota synchronization, provider-specific business logic, sophisticated adaptive optimization, unrelated changes to DSH.

Open Design Questions

1. Should the router live inside DSH initially, or immediately as a standalone npm package?
2. What should the canonical Source interface look like?
3. Should health/quota state be pushed into the router or queried through an interface?
4. Should task profiles be static configuration or support runtime registration?
5. What package ownership/release model would be preferred?
6. Should the first PR target only the routing abstraction, with npm publishing handled separately?

Proposed Implementation Plan

Phase 1 - Design validation: agree on core interfaces, package boundaries, ownership/release expectations.

Phase 2 - Initial implementation: implement policy engine, source abstraction, health-aware fallback, deterministic tests; keep provider-specific integrations behind adapters.

Phase 3 - Integration: integrate with DSH where appropriate; validate the public API against real routing scenarios.

Phase 4 - Packaging: if maintainers agree, publish the routing layer as an npm package, document the API, establish release/versioning process.

Success Criteria

The initial implementation should demonstrate that routing policy is independent from provider implementation, unhealthy sources are skipped correctly, fallback behavior is deterministic, adding a new source does not require rewriting routing policy, and the public API is small enough to be reused outside the immediate integration.

Ownership / Contribution

I would be happy to take the lead on the initial implementation and iterate based on maintainer feedback.

---

Disclosure: I maintain free-best-router, an OpenAI-compatible router across free AI providers. The proposal is informed by what works (and what does not) in that codebase; it is a generalization, not a rebrand. Happy to ship the reference implementation as a standalone npm package, a DSH plugin, or both, depending on maintainer direction.