Skip to content

[Bug] Custom providers with secretref-managed apiKey fail with 503 auth_unavailable after update to 2026.6.5 #92097

Closed

Description

@LINSUISHENG034

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

After updating to v2026.6.5, custom providers in models.json that use apiKey: "secretref-managed" (non-env SecretRef backed by openclaw-config-secrets.json) fail with 503 auth_unavailable: no auth available (providers=codex, model=gpt-5.5). This makes all agents using custom providers with openai-responses API completely non-functional — every request returns the "Something went wrong" error to end users.

Steps to reproduce

  1. Configure a custom provider in models.json with api: "openai-responses" and apiKey: "secretref-managed" (file-backed SecretRef from openclaw-config-secrets.json).
  2. Set the corresponding provider API key in ~/.openclaw/credentials/openclaw-config-secrets.json under openclaw.models.providers.<providerId>.apiKey.
  3. Run openclaw update to update to v2026.6.5.
  4. Restart the gateway.
  5. Send a request to an agent that uses this custom provider.

Expected behavior

The gateway's secrets runtime resolves the file-backed SecretRef at startup and makes the actual API key available to the auth resolution path. Custom providers with openai-responses API should work the same as they did before the update.

Actual behavior

The request fails with:

503 auth_unavailable: no auth available (providers=codex, model=gpt-5.5)

Gateway log shows:

{
  "subsystem": "openai-transport",
  "message": "[responses] error provider=cliproxyapi api=openai-responses model=gpt-5.5 name=Error status=503 code=internal_server_error type=server_error message=503 auth_unavailable: no auth available (providers=codex, model=gpt-5.5)"
}

The resolveUsableCustomProviderApiKey() function in src/agents/model-auth.ts returns null for non-env secretref-managed markers. The fallback resolveProviderSyntheticRuntimeAuth() also returns undefined for custom providers without matching plugins (no resolveSyntheticAuth hook). The runtime config snapshot that contains the resolved values is never consulted for these providers.

Root cause analysis

The auth resolution chain in resolveApiKeyForProvider() (model-auth.ts) has a gap:

  1. resolveUsableCustomProviderApiKey() — for apiKey: "secretref-managed", isNonSecretApiKeyMarker() returns true, and the function returns null. It does NOT check the runtime config snapshot where the secrets runtime has already resolved the actual value.

  2. resolveProviderSyntheticRuntimeAuth() — for custom providers without a matching plugin, resolveProviderSyntheticAuthWithPlugin() returns undefined. The function then returns {} at the !directAuth check, never reaching the getRuntimeConfigSnapshot() fallback that would have the resolved value.

  3. The gateway's secrets runtime (prepareSecretsRuntimeSnapshot) correctly resolves file-backed SecretRefs and stores actual values in the runtime snapshot via setRuntimeConfigSnapshot(). But the auth resolution path for custom providers never consults this snapshot.

This is a regression — before v2026.6.5, the openai-responses API path resolved auth through a different code path that successfully read the resolved snapshot. The v2026.6.5 routing changes (likely related to OpenAI provider unification / codex routing) changed the auth resolution flow, introducing this gap.

Workaround

Replace apiKey: "secretref-managed" in models.json with the environment variable name that exists in gateway.systemd.env:

{
  "cliproxyapi": {
    "apiKey": "CLIPROXYAPI_API_KEY",
    ...
  }
}

Env var references are resolved by resolveUsableCustomProviderApiKey() via process.env and work correctly.

OpenClaw version

2026.6.5 (5181e4f)

Operating system

Ubuntu 22.04 LTS (Linux 5.15.0-131-generic)

Install method

npm global install (openclaw update)

Model

cliproxyapi/gpt-5.5 (custom provider, openai-responses API)

Provider / routing chain

cliproxyapi → openai-responses API → codex transport subsystem → 503 auth_unavailable

Related issues

Activity

  1. clawsweeper commented on Jun 11, 2026

    @clawsweeper
    Contributor

    Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed June 11, 2026, 12:00 PM ET / 16:00 UTC.

    Summary
    Codex review failed: retryable codex transport failure (exit 1).

    Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path.

    Ways to help us reproduce this

    • Add a screenshot or short recording showing the behavior.
    • Include the exact command, prompt, or workflow that triggered it.
    • Add expected vs actual behavior.

    Next step
    Review did not complete, so no work-lane recommendation was made.

    Review details

    Best possible solution:

    Retry the Codex review after fixing the execution failure.

    Do we have a high-confidence way to reproduce the issue?

    Unclear. The review failed before ClawSweeper could establish a reproduction path.

    Is this the best way to solve the issue?

    Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction.

    AGENTS.md: unclear because the file could not be read completely.

    Remaining risk / open question:

    • No close action taken because the review did not complete.

    Codex review notes: model internal, reasoning high; reviewed against 3d6252a51772.

    Label changes

    Label changes:

    • add issue-rating: 🦪 silver shellfish: Current issue advisory state selects this label.
    • add clawsweeper:needs-info: Current issue advisory state selects this label.
    • remove P1: Current review triage priority is none.
    • remove clawsweeper:fix-shape-clear: Current issue advisory state no longer selects this label.
    • remove clawsweeper:queueable-fix: Current issue advisory state no longer selects this label.
    • remove clawsweeper:source-repro: Current issue advisory state no longer selects this label.
    • remove impact:auth-provider: Current review selected no impact labels.
    • remove issue-rating: 🦞 diamond lobster: Current issue advisory state no longer selects this label.
    Evidence reviewed

    What I checked:

    • failure reason: retryable codex transport failure.
    • codex failure detail: Codex review failed for this issue with exit 1.
    • codex stderr: tRuntimeConfigSnapshot } from "../config/config.js";.
    • codex stdout: No stdout captured.

    Likely related people:

    • unknown: Codex failed before it could trace repository history. (role: review did not complete; confidence: low)
    How this review workflow works
    • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
    • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
    • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
    • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
    • Maintainers can also comment @clawsweeper review to request a fresh review only.
    • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
    • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
    • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
  2. added
    P1High-priority user-facing bug, regression, or broken workflow.
    impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    on Jun 11, 2026
  3. wangmiao0668000666 commented on Jun 11, 2026

    @wangmiao0668000666
    Contributor

    I'd like to work on this.

    The root cause is clearly identified in the issue: returns at the check without consulting , which contains the resolved SecretRef value for custom providers with . I'll implement the fix in and add a regression test.

  4. wangmiao0668000666 commented on Jun 11, 2026

    @wangmiao0668000666
    Contributor

    I'd like to work on this. I've identified the root cause in resolveProviderSyntheticRuntimeAuth and opened PR #92113 with a fix and regression test.

  5. self-assigned this
    on Jun 11, 2026
  6. joshavant commented on Jun 12, 2026

    @joshavant
    Collaborator

    Thanks @LINSUISHENG034 for the detailed report and root-cause notes.

    This is fixed by #92235, merged as 9386d62. The failure was that custom provider entries with managed non-env SecretRefs could remain as secretref-managed in the source config, while the auth resolver did not bridge that marker to the already-resolved runtime config snapshot. That made affected custom providers look unauthenticated and produced the 503 auth_unavailable path you reported.

    The merged fix resolves explicit managed provider SecretRefs from the matching runtime snapshot, keeps explicit provider credentials ahead of ambient profile/env auth, and prevents synthetic local/plugin auth from shadowing those configured credentials. We also verified the exact merged-head behavior with Crabbox: file-backed SecretRef custom providers reached upstream with Authorization present, and marker-only config without a resolved runtime value failed before making a request.

    Closing as fixed. Thanks again for the clear report.

  7. locked as resolved and limited conversation to collaborators on Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions