hermes auth add openai-codex writes only to credential_pool; runtime resolver reads providers.tokens — fails with 'missing access_token'

[julio-cloudvisor]

Summary

hermes auth add openai-codex writes the OAuth credential to the credential pool (auth_store["credential_pool"]["openai-codex"][*]), but the runtime resolver for openai-codex reads from the legacy single-active provider state (auth_store["providers"]["openai-codex"]["tokens"]). The two storage layers are not synced by the auth add codepath, so after a successful hermes auth add openai-codex the runtime fails with:

Provider authentication failed: Codex auth is missing access_token.
Run `hermes auth` to re-authenticate. Run `hermes model` to re-authenticate.

The error message correctly suggests two different commands because the two storage layers exist — but a user who runs only hermes auth add openai-codex (the documented command for adding pooled credentials per hermes auth --help) ends up in a state where the credential is visibly present but unusable.

Repro

Tested against hermes-agent HEAD (commit one ahead of 7cd1f6e2e from PR #30973, on a long-running deployed gateway).

1. Start with an OpenAI Codex provider configured on a profile: model.provider = openai-codex, model.model = gpt-5.5. (No prior auth required for this repro, but in practice the profile typically had earlier auth state that has since expired — leaving providers.openai-codex.tokens as an empty dict with last_auth_error populated.)

2. Run:

   hermes auth add openai-codex
   

3. Complete the device-code flow successfully. Output:

   Added openai-codex OAuth credential #N: "openai-codex-oauth-N"
   

4. Verify the pool has the credential:

   hermes auth list
   

   Shows the new entry with auth_type=oauth, source=device_code.

5. Inspect ~/.hermes/auth.json (or ~/.hermes/profiles/<profile>/auth.json when profile-scoped):

   • credential_pool["openai-codex"][N-1] has top-level access_token (2007-char string) and refresh_token (90-char string) — both valid and freshly issued.
   • providers["openai-codex"]["tokens"] is still {} (empty dict — access_token and refresh_token both absent).

6. Make any API call that triggers the openai-codex resolver (e.g. send a chat message, or have a gateway worker invoke a tool that calls the model). The resolver fails with Codex auth is missing access_token.

Root cause

The runtime resolver resolve_codex_runtime_credentials() in hermes_cli/auth.py:3348 calls _read_codex_tokens():

def resolve_codex_runtime_credentials(...) -> Dict[str, Any]:
    data = _read_codex_tokens()
    tokens = dict(data["tokens"])
    access_token = str(tokens.get("access_token", "") or "").strip()
    ...

_read_codex_tokens() (line 3126) reads exclusively from providers["openai-codex"]:

def _read_codex_tokens(*, _lock: bool = True) -> Dict[str, Any]:
    ...
    state = _load_provider_state(auth_store, "openai-codex")
    ...
    tokens = state.get("tokens")
    ...
    access_token = tokens.get("access_token")
    if not isinstance(access_token, str) or not access_token.strip():
        raise AuthError(
            "Codex auth is missing access_token. Run `hermes auth` to re-authenticate.",
            ...
        )

_load_provider_state (line 1077) reads auth_store["providers"][provider_id]. It does not consult auth_store["credential_pool"][provider_id].

The legacy login path _login_openai_codex in hermes_cli/auth.py:6232 (reachable via hermes model → pick openai-codex → Reauthenticate) does the right thing: it calls _codex_device_code_login() followed by _save_codex_tokens(creds["tokens"], ...), which writes to providers["openai-codex"]["tokens"] via _save_provider_state (line 1085). This is the working path.

The pool-add path in hermes_cli/auth_commands.py:313+ does NOT call _save_codex_tokens:

if provider == "openai-codex":
    auth_mod.unsuppress_credential_source(provider, "device_code")
    creds = auth_mod._codex_device_code_login()
    label = ...
    entry = PooledCredential(
        provider=provider,
        ...,
        access_token=creds["tokens"]["access_token"],
        refresh_token=creds["tokens"].get("refresh_token"),
        ...
    )
    pool.add_entry(entry)              # ← writes to credential_pool only
    print(f'Added {provider} OAuth credential #...')
    return                              # ← returns without populating providers[openai-codex][tokens]

So both paths invoke the same device-code flow and obtain the same creds["tokens"], but only the legacy path persists them to the location the runtime resolver reads from.

Expected behaviour

After hermes auth add openai-codex returns successfully, the runtime resolver should be able to use the newly-issued credential. Options for the fix, in roughly increasing order of scope:

1. Minimal fix: in hermes_cli/auth_commands.py:313+, after pool.add_entry(entry), also call auth_mod._save_codex_tokens(creds["tokens"], creds.get("last_refresh")). This keeps the pool authoritative for hermes auth list while also populating the legacy state so the resolver succeeds. Same shape _login_openai_codex already uses.

2. Resolver fix: make resolve_codex_runtime_credentials() (and _read_codex_tokens()) fall back to the credential pool when providers["openai-codex"]["tokens"] is missing or empty. Pick the highest-priority entry (or the one marked active) from credential_pool["openai-codex"] and use its top-level access_token / refresh_token. This is the more durable fix — it makes the two storage layers genuinely interchangeable.

3. Schema unification: deprecate the legacy providers["openai-codex"]["tokens"] block entirely; require all openai-codex credential consumers to go through the pool. Bigger change, breaking for any out-of-tree resolver, but eliminates the split-brain.

Workaround for users

For anyone hitting this in production:

hermes -p <profile> model
# Pick: openai-codex
# Pick: 2. Reauthenticate (new OAuth login)
# Complete the device-code flow

This invokes the legacy _login_openai_codex path which calls _save_codex_tokens, populating providers["openai-codex"]["tokens"] correctly. The runtime resolver then resolves cleanly. Restart the gateway.

The credential pool entries added by the earlier failed hermes auth add are harmless — they remain in the pool but aren't read by the openai-codex resolver. They can be removed with hermes auth remove openai-codex-oauth-<N> if pool hygiene matters.

Why this matters

The two-command suggestion in the error message (Run hermes auth … Run hermes model …) is a tell that the team is already aware of the split, but the auth add command's success message gives users no signal that runtime resolution will still fail. Users following the hermes auth --help documentation path land in the broken state. Documentation alone can't fix this — hermes auth add openai-codex either needs to populate both storage layers, or the error message needs to be more directive ("Run hermes model to complete the openai-codex login. hermes auth add writes only to the credential pool and does not register the runtime credential for this provider.").

Environment

• hermes-agent: HEAD as of 2026-05-26 (one commit ahead of PR feat(kanban): opt-in HERMES_KANBAN_SYNCHRONOUS_MODE for synchronous=FULL durability hardening #30973 / commit 7cd1f6e2e).
• Python 3.12, Linux x86_64.
• Two profile auth.json files inspected (one global, one profile-scoped) — both show the same split-brain shape post-auth add.

Happy to send a PR for fix (1) — minimal change, mirrors _login_openai_codex's persistence path. (2) is the better long-term fix but a larger diff that touches the resolver semantics, and would benefit from team review on whether the pool ordering becomes the canonical "active credential" signal.