OpenAI Codex usage reset leaves stale model_rate_limits and causes fake 429/503 no-available-accounts

[stgmt]

Symptom

After the real ChatGPT/Codex usage window reset, the UI showed quota headroom again, but the proxy kept rejecting Claude Code requests with local account-selection failures such as:

• Request rejected (429) · All configured accounts that support "gpt-5.5" are rate-limited ...
• API Error: 503 Service temporarily unavailable
• gateway log: openai_messages.account_select_failed / error: "no available accounts"

This was reproducible even though direct quota state was no longer exhausted.

Evidence from local Docker runtime

Runtime: sub2api-codex Docker compose on port 18081.

Before recovery, the account row had stale quota state in accounts.extra, including old Codex usage and model cooldowns. Manually clearing stale codex_*used_percent and quota-origin model_rate_limits made requests work again without changing the upstream account.

After the code fix and image rebuild, the live DB state is:

id=1 platform=openai status=active codex_5h_used_percent=2 codex_7d_used_percent=0
model_rate_limits={"gpt-5.6-luna":{"reason":"upstream_404_model_not_found",...}}

The quota-origin fake cooldowns for gpt-5.5, gpt-5.6-sol, and gpt-5.6-terra are gone. The non-quota gpt-5.6-luna upstream-404 cooldown is intentionally preserved.

Live probes through /v1/messages after redeploy:

• gpt-5.5: HTTP 200
• gpt-5.6-sol: HTTP 200
• gpt-5.6-terra: HTTP 200
• gpt-5.6-luna: local 503 from preserved upstream_404_model_not_found, not a quota cooldown

Root Cause

sub2api persisted OpenAI/Codex model cooldowns in accounts.extra.model_rate_limits. When OpenAI/Codex quota reset outside the proxy, stale openai_model_rate_limited entries could survive and account selection filtered the account before any upstream request had a chance to refresh the real usage headers.

The existing Codex usage snapshot update path wrote fresh codex_* usage fields, but did not clear obsolete quota-origin model_rate_limits. That made the proxy report fake 429/503 errors after quota was already available again.

Fix in fork

When a fresh Codex usage snapshot/probe shows quota headroom, clear only quota-origin OpenAI model cooldowns:

• remove reason == openai_model_rate_limited
• remove reason prefixed with openai_model_rate_limited:
• preserve upstream model-not-found cooldowns, image cooldowns, and unknown reasons

The recovery is wired into both:

• live gateway usage-header snapshots
• account usage/probe snapshot persistence

Tests

Added regression coverage for:

• detecting Codex quota headroom from fresh usage headers
• preserving non-quota cooldown reasons
• clearing stale quota cooldowns after a fresh non-exhausted snapshot
• not clearing quota cooldowns when a fresh snapshot still reports an exhausted window

Verification command:

go test ./internal/service -count=1

Result:

ok github.com/Wei-Shaw/sub2api/internal/service 51.057s

Docker verification

Rebuilt and redeployed the local image:

docker build -t sub2api-codex:local-token-usage .
docker compose -p sub2api-codex up -d sub2api

Container health after deploy:

sub2api-codex Up (healthy) 0.0.0.0:18081->8080/tcp
/health => {"status":"ok"}

[stgmt]

Fixed in branch fix/openai-messages-empty-stream-errors at commit a31dd75.

Verification after the fix:

• go test ./internal/service -count=1 -> passed
• rebuilt local Docker image sub2api-codex:local-token-usage
• redeployed sub2api-codex compose service
• /health reports ok
• live /v1/messages probes through the proxy returned HTTP 200 for gpt-5.5, gpt-5.6-sol, and gpt-5.6-terra
• DB now preserves only the non-quota gpt-5.6-luna upstream_404_model_not_found cooldown; quota-origin fake cooldowns were cleared

[stgmt]

Done and merged to main.

Main now contains:

• a31dd750 — stale Codex quota/model cooldown recovery for fake 429/503 no-available-accounts
• 0c2860f1 — merge compatibility fix after syncing current stgmt/main

Verified after merge:

• go test ./internal/service -count=1 -> passed
• focused stale-cooldown regression tests -> passed
• local Claude Code/sub2api config switched to GPT-5.6 profile
• live /v1/messages probes returned 200 for gpt-5.6-sol, gpt-5.6-terra, claude-opus-4-8, claude-sonnet-4-6, and claude-haiku-4-5
• usage_logs confirmed mappings:
  • claude-opus-4-8 -> gpt-5.6-sol
  • claude-sonnet-4-6 -> gpt-5.6-terra
  • claude-haiku-4-5 -> gpt-5.6-terra

Skill sub2api-claude-code-codex was updated and validated locally for the GPT-5.6 profile and stale cooldown self-heal workflow.