[BUG] ollama-cloud per-model subscription 403 cools down the whole connection (free models knocked out)

[diegosouzapw]

Bug

Source: Discussion #3001 by @paraflu

A per-model subscription 403 from Ollama Cloud (e.g. deepseek-v4-pro → 403 "this model requires a subscription, upgrade for access") cools down the entire ollama-cloud connection instead of locking out only the paid model. While the connection is in cooldown, the free models on the same key (e.g. gemma4:31b) also become temporarily unavailable — and repeated hits on the paid model escalate the cooldown via exponential backoff.

This matches the reporter's symptom: "any ollamacloud model gave me this answer" — after the paid model's 403, the whole connection is cooled down, so subsequent requests to free models fail too until it recovers.

Symptom (reporter's words)

Any ollamacloud model gave me this answer
Error: [ollama-cloud/deepseek-v4-pro] [403]: this model requires a subscription, upgrade for access: https://ollama.com/upgrade (ref: ...) (reset after 1s)
But deepseek is an available model... https://ollama.com/library/deepseek-v4-pro

(The 403 itself is correct — deepseek-v4-pro is an upstream Ollama Cloud paid-tier model. The bug is the blast radius of OmniRoute's reaction to it.)

Root cause (verified, file:line)

ollama-cloud is registered with passthroughModels: true and authType: "apikey":

• open-sse/config/providerRegistry.ts:3129 — passthroughModels: true, authType: "apikey".
• So hasPerModelQuota("ollama-cloud", ...) returns true (open-sse/services/accountFallback.ts:511, passthrough branch).

A passthrough provider multiplexes many upstream models behind one connection, so a per-model failure should stay model-scoped. But the 403 escapes every model-lockout path:

1. classifyProviderError returns null for this 403.
   open-sse/services/errorClassifier.ts:146 — a 403 on an apikey-category provider returns null unless the body matches isCreditsExhausted(...). The message "this model requires a subscription, upgrade for access" matches none of CREDITS_EXHAUSTED_SIGNALS (open-sse/services/accountFallback.ts:116 — insufficient_quota, out of credits, payment required, etc.). So it is not classified as QUOTA_EXHAUSTED, and the chatCore per-model lockout path (open-sse/handlers/chatCore.ts:4574, gated on errorType === QUOTA_EXHAUSTED) never fires.

2. The markAccountUnavailable model-lockout gate excludes 403.
   src/sse/services/auth.ts:1684-1689 — the per-model lockout branch is gated on (status === 404 || status === 429 || status >= 500). 403 is not in the set, so even though isPerModelQuotaProvider is true, it is skipped.

3. The only 403 → model-lockout special case is hard-coded to grok-web.
   src/sse/services/auth.ts:1732 — if (provider && resolveProviderId(provider) === "grok-web" && status === 403 && model) → "Mode-only lockout … connection stays active". ollama-cloud does not match.

4. Fallthrough: connection-wide cooldown.
   The 403 reaches the generic api-key branch in checkFallbackError (open-sse/services/accountFallback.ts:1338-1346) → buildRetryableFallback(RateLimitReason.AUTH_ERROR), a connection-level retryable cooldown with exponential backoff (baseCooldownMs * 2 ** failureIndex). The whole ollama-cloud connection is now in cooldown — free models included.

Proposed fix

Generalize the grok-web 403 precedent to all per-model-quota providers: a model/permission-scoped 403 on a passthrough/hasPerModelQuota provider should produce a model lockout, not a connection cooldown.

Concretely, one of (prefer the first — narrowest, matches the existing pattern):

• In src/sse/services/auth.ts, extend the per-model lockout gate (line ~1689) to include 403 when isPerModelQuotaProvider is true (and the 403 is not a clear whole-key auth failure — i.e. not accountDeactivated / not an AUTH_CREDENTIAL_ERROR_PATTERNS match), mapping it to reason "forbidden" / "model_locked". This mirrors the grok-web branch at line 1732 but keyed on hasPerModelQuota instead of a single provider id.
• Keep terminal 403s (banned/deactivated key) on their existing path — only model/permission/subscription-scoped 403s become model lockouts.

Net effect: hitting deepseek-v4-pro's subscription 403 locks only ollama-cloud/deepseek-v4-pro for the short reset window; gemma4:31b and the other free models on the same connection keep serving.

TDD requirement (please implement test-first)

Per the repo's TDD policy, write the failing test before the fix:

1. Reproduction test (must fail on current main/release): simulate a 403 with body "this model requires a subscription, upgrade for access" for provider="ollama-cloud", model="deepseek-v4-pro" on a connection. Assert:
   • The connection's rateLimitedUntil / testStatus is NOT set (connection stays eligible).
   • A model lockout exists for ollama-cloud/deepseek-v4-pro (e.g. via getModelLockout(...) / isModelLockedOut(...) from open-sse/services/accountFallback.ts).
   • A second model on the same connection (ollama-cloud/gemma4:31b) is still eligible (not locked, not cooled down).
2. Regression guard: a genuine whole-key 403 (account deactivated / invalid key) still cools down / deactivates the connection (must NOT be downgraded to a model lockout).
3. Backoff guard: repeated subscription-403s on the paid model do not escalate a connection-wide backoff.

Suggested test location: tests/unit/ alongside the existing accountFallback / auth resilience tests. Then implement the gate change and confirm the suite (including npm run test:coverage, lint, typecheck, build) is green.

Current workaround (for users today)

Use Ollama Cloud's free-tier models in your combo (e.g. gemma4:31b) and avoid mixing paid-tier models (deepseek-v4-pro, kimi-k2.6, etc.) into the same combo unless your Ollama account has the matching subscription. If you only hit paid models occasionally, the connection self-recovers after the short cooldown window (the (reset after Ns) in the error).

Additional context

• Discussion: ollamacloud #3001
• Related (closed, different paths): [BUG] Some Ollama Cloud models returning 404 Not Found #2203 (ollama-cloud 404 — already covered by the per-model lockout gate since 404 is in the set), [BUG] Ollama provider is not working error ([401]: unauthorized) #1262 / ollama cloud 401 error #276 (ollama-cloud 401).
• Resilience design references: docs/architecture/RESILIENCE_GUIDE.md (Model Lockout vs Connection Cooldown scopes) and the project CLAUDE.md resilience section ("Provider-specific mode/model permission failures … Model lockout lets the same connection continue serving other models").

[diegosouzapw]

Fixed in release/v3.8.9 (#3095). A per-model subscription/permission 403 from a passthrough provider (ollama-cloud) now locks out only the failing model instead of cooling the whole connection — so the free models on the same key keep serving, and repeated paid-model 403s no longer escalate a connection-wide backoff. Terminal 403s (banned/deactivated key) still deactivate the connection. Merged into the active development branch; ships in v3.8.9.