[Bug]: Codex usage_limit_reached 429 retries exhausted pool credential once before rotating

[Qwinty]

Bug Description

OpenAI Codex can return HTTP 429 with a structured error body like:

{
  "error": {
    "type": "usage_limit_reached",
    "message": "The usage limit has been reached"
  }
}

Hermes currently extracts only error.code / error.error into the credential-pool error context, so error.type=usage_limit_reached is lost. The central classifier sees HTTP 429 and treats it as a generic transient rate_limit.

That makes _recover_with_credential_pool() keep the same exhausted Codex credential on the first failure and only rotate after a second 429 in the same retry sequence. For Codex account usage windows this is not useful: the current OAuth profile is exhausted until reset, while other pool entries may still have available quota.

Steps to Reproduce

1. Configure openai-codex with two or more OAuth entries in the credential pool.
2. Use an entry whose Codex quota is exhausted while another pool entry still has quota.
3. Trigger a Codex request that returns HTTP 429 with error.type = "usage_limit_reached".
4. Observe the first recovery attempt retrying the same exhausted credential instead of marking it exhausted and rotating immediately.

A minimal unit repro is _recover_with_credential_pool(status_code=429, has_retried_429=False, error_context={"reason": "usage_limit_reached"}): today it returns (False, True) instead of rotating.

Expected Behavior

A Codex usage_limit_reached 429 should be treated as account/window exhaustion for pool recovery. Hermes should mark that pool entry exhausted and rotate to the next available same-provider credential immediately.

Actual Behavior

Hermes treats the response as a generic transient 429, retries the same credential once, and can appear stuck on an exhausted account even when hermes auth list openai-codex shows other accounts without limits.

Affected Component

Agent Core (conversation loop, context compression, memory)

Messaging Platform (if gateway-related)

Telegram and CLI can both hit the same core recovery path; the bug is in run_agent.py.

Debug Report

N/A for this report. This was reduced to the recovery helper and covered with unit tests in the companion PR.

Operating System

Ubuntu 24.04

Python Version

Python 3.11+

Hermes Version

main at d541628

Additional Logs / Traceback (optional)

Known provider message:

HTTP 429: The usage limit has been reached

Known structured body field that is currently dropped:

error.type = usage_limit_reached

Root Cause Analysis (optional)

Two details combine:

1. AIAgent._extract_api_error_context() ignores payload["type"], so OpenAI's usage_limit_reached type does not reach the pool recovery code.
2. _recover_with_credential_pool() treats all 429s as transient FailoverReason.rate_limit, where the first 429 intentionally retries the same credential.

That first retry makes sense for throttling, but not for Codex usage_limit_reached, which is a quota-window exhaustion signal for the current account.

Similar Issues / PRs Checked

This is intentionally narrower than the currently open related work:

• fix(credential-pool): rotate immediately on 429 when credential already exhausted #26149 rotates immediately only when the current entry is already marked exhausted; it does not handle a fresh usage_limit_reached body on the first 429.
• fix: keep credential pool aligned across fallback runtime switches #25733 aligns credential pools with runtime/fallback state; it does not change the usage_limit_reached classification/recovery behavior.
• fix: load credential pool in switch_model for new provider (#25273) #25277 loads pools during /model switch; different entry path.
• fix(codex): credential pool rotation on Responses API soft failures #24173 handles Codex Responses soft failures where HTTP status is 200 with response.status=failed; this issue is the HTTP exception path.
• fix(auth): preserve concurrently-added credentials on pool rewrite #19608 fixes concurrent auth-store stale writes; different persistence race.
• bug(auxiliary): pooled Codex compression can stay pinned to exhausted auth after rotation #22778/fix(auxiliary): rotate pooled auth after quota failures (salvage #22779) #22792 fixed auxiliary-client cache pinning; this is the main agent recovery path.

Searches included usage_limit_reached with first 429, second 429, retry same credential, rotate immediately, and The usage limit has been reached + credential pool across open and closed issues/PRs.

Proposed Fix (optional)

• Extract payload["type"] as the context reason when code is absent.
• In rate-limit pool recovery, if error_context.reason contains usage_limit_reached or the message contains usage limit has been reached, skip the first-same-credential retry and immediately call mark_exhausted_and_rotate().
• Add regression tests for both context extraction and immediate rotation.

Are you willing to submit a PR for this?

Yes; companion PR follows.

[Qwinty]

Follow-up: the immediate-rotation fix described here handles the first usage_limit_reached 429, but a live pool investigation found another related stale-state path.

Even after rotation is immediate, Hermes can keep an openai-codex pool entry marked last_status=exhausted until the stored reset timestamp while the live Codex usage endpoint already reports that same entry as allowed=true. That makes hermes auth list openai-codex and runtime selection produce false-positive exhausted pools.

The additional fix I validated locally is to reconcile exhausted Codex pool entries against live per-entry usage before selection/listing, and to pass the selected credential_pool_entry_id into recovery so the exact failed entry is marked rather than relying on mutable pool.current().