Skip to content

Copilot provider auto-derived from a classic ghp_* GITHUB_TOKEN creates a permanently-failing fallback in the credential pool #47708

Closed

Description

@tcconnally

Summary

When a GITHUB_TOKEN is present in the environment (commonly set for gh / the GitHub MCP, and here injected via Bitwarden Secrets Manager), Hermes auto-registers a copilot credential in the gateway credential pool, pointed at https://api.githubcopilot.com. If that token is a classic ghp_* PAT, the Copilot API rejects every request, so the copilot entry is a permanently-broken fallback that gets tried (and fails) on every failover.

Observed

auth.json credential pool entry, auto-created with no explicit Copilot setup:

"copilot": [
  {
    "label": "GITHUB_TOKEN",
    "auth_type": "api_key",
    "source": "env:GITHUB_TOKEN",
    "base_url": "https://api.githubcopilot.com",
    "secret_source": "bitwarden"
  }
]

Runtime failures (hermes-agent 0.16.0):

Token from GITHUB_TOKEN is not supported: Classic Personal Access Tokens (ghp_*)
are not supported by the Copilot API. Use one of:
  → `copilot login` or `hermes model` to authenticate via OAuth
  → A fine-grained PAT (github_pat_*) with Copilot Requests permission
  → `gh auth login` with the default device code flow (produces gho_* tokens)
API call failed: BadRequestError [HTTP 400]
   Provider: copilot  Model: gpt-5.x
   Error: Personal Access Tokens are not supported for this endpoint

Impact

  • A GITHUB_TOKEN set for unrelated reasons (gh CLI, GitHub MCP) silently enrolls a Copilot provider the user never configured.
  • On any primary-model failure, failover burns a hop on copilot and 400s, adding log noise and latency.
  • Confusing UX: users see copilot/gpt-5.x being attempted and reasonably ask "why is Copilot my fallback?" when they never set it up.

Suggested fix

Any one (or combination) of:

  • Don't auto-derive a Copilot credential from classic ghp_* tokens — only enroll gho_* (device-flow) or github_pat_* fine-grained tokens that carry Copilot Requests permission.
  • Mark the credential unhealthy after the first auth rejection so failover skips it instead of retrying every turn (the error is deterministic, not transient).
  • Make Copilot auto-derivation from GITHUB_TOKEN opt-in.

The token-type is already detectable (the error message above distinguishes ghp_* / github_pat_* / gho_*), so this can be gated before the credential is ever added to the pool.


Filed via Claude Code while diagnosing an unexpected copilot/gpt-5.x fallback on hermes-agent 0.16.0.

Activity

  1. added
    type/bugSomething isn't working
    comp/gatewayGateway runner, session dispatch, delivery
    area/authAuthentication, OAuth, credential pools
    P2Medium — degraded but workaround exists
    on Jun 17, 2026
  2. alt-glitch commented on Jun 17, 2026

    @alt-glitch

    Related: #12650 (Copilot token validation accepts arbitrary strings) and #35946 (spurious Copilot WARNING when GITHUB_TOKEN is set but Copilot unconfigured). This issue adds a distinct angle: don't auto-derive a Copilot credential from classic ghp_* tokens at all, and mark the credential unhealthy after the first deterministic auth rejection so failover skips it.

  3. Ben-Home commented on Jun 17, 2026

    @Ben-Home

    This is a configuration detection issue -- Hermes sees a GITHUB_TOKEN env var and assumes it's a valid Copilot credential without checking the token type first.

    The fix should be a pre-validation step in the credential auto-registration:

    1. Check token prefix before registering: ghp_* (classic PAT) -> skip Copilot registration entirely. github_pat_* (fine-grained) -> check for copilot scope. gho_* (OAuth token) -> check scopes.

    2. Make auto-registration opt-in, not default: The current behavior silently registers a credential that fails 100% of the time for ghp_* tokens. A config flag like auto_register_copilot: false (default) would prevent this entirely. Users who want Copilot can set it to true after verifying their token type.

    3. Validate at registration time, not at call time: A quick GET to the Copilot API user endpoint with the token would catch the classic PAT rejection immediately and log a clear warning instead of silently adding a broken credential to the pool.

    The broader issue is that Hermes treats GITHUB_TOKEN as a universal GitHub credential, but Copilot has a completely different auth domain. Environment variable presence does not equal capability availability.

  4. scottymcraig commented on Jun 18, 2026

    @scottymcraig

    Adding the workaround that got this machine unstuck, in case it helps other users or points at the right fix.

    In our setup, OAuth would not complete while the existing GitHub token/gh-derived auth path was visible, because Hermes kept treating that token as the available Copilot credential. The workaround was to temporarily hide the pre-existing GitHub auth source, run the OAuth flow, then restore normal GitHub CLI/git auth afterwards.

    What worked:

    1. Temporarily remove the token source from the Hermes setup environment:
    • unset/move aside GITHUB_TOKEN if it is injected by the shell/secrets manager, and/or
    • temporarily move/hide the gh path/config that Hermes is auto-detecting.
    1. Run the Hermes Copilot OAuth setup again, e.g. through hermes model / hermes auth add ..., so it is forced to create a real OAuth credential instead of reusing the PAT-derived one.
    2. Restore gh / the GitHub token source after OAuth succeeds.
    3. Re-run GitHub CLI git credential wiring if needed:
    gh auth setup-git
    gh auth status
    gh api user --jq .login
    

    After doing that here, gh auth status showed the expected logged-in account and git HTTPS operations were wired through !/usr/bin/gh auth git-credential.

    The important takeaway is that the existing token needed to be hidden only during OAuth enrollment. Once the OAuth credential existed, normal gh/git auth could be restored. That suggests the setup path should distinguish "token present" from "valid Copilot OAuth credential present" and still allow explicit OAuth enrollment even when a GitHub token or gh credential already exists.

  5. added a commit that references this issue on Jul 14, 2026
    74fdf79
  6. steininger commented on Jul 24, 2026

    @steininger

    Corroborating this issue with a sanitized second-order failure observed on Hermes v0.18.2.

    Abstract pool shape:

    priority 0: valid Copilot OAuth/device credential
    priority 1+: classic GitHub PATs auto-seeded from generic GitHub auth sources
    

    Failure chain:

    valid OAuth credential receives a non-auth provider 403
    → credential pool marks it exhausted
    → rotation selects a classic PAT
    → Copilot rejects the PAT with HTTP 400
    → the PAT error becomes the visible symptom and masks the original 403
    

    Because pool state is persisted, restarting the gateway does not remove this chain. An explicit Copilot auth reset temporarily restores the valid OAuth entry, but the failure recurs when the original model/route error is classified as credential exhaustion.

    Recommended acceptance coverage:

    • ghp_* credentials are never enrolled into the Copilot pool from GH_TOKEN, GITHUB_TOKEN, or gh auth token;
    • an explicitly enrolled Copilot OAuth credential remains higher priority than generic GitHub credentials;
    • deterministic model/endpoint errors do not exhaust the Copilot credential;
    • when rotation does occur, diagnostics retain both the original provider error and the rejected fallback source, without logging secrets.

    No private repository, account, hostname, or credential data is included here.

  7. added
    comp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint
    sweeper:risk-security-boundarySweeper risk: may affect sandboxing, auth, credentials, or sensitive data
    and removed
    comp/gatewayGateway runner, session dispatch, delivery
    sweeper:risk-security-boundarySweeper risk: may affect sandboxing, auth, credentials, or sensitive data
    on Jul 24, 2026
  8. GottZ commented on Aug 3, 2026

    @GottZ

    This was generated by AI during triage.

    Summary

    Nineteen PRs address or reference this credential-resolution complex, but only #47718 and #47770 directly fix #47708 by preventing classic ghp_* tokens from being auto-seeded as Copilot credentials; #20368 touches a separate Copilot pool-resolution path, while the other diffs concern credential discovery, precedence, exhaustion, or profile isolation. #47718 is the best existing fix because it reuses Copilot token validation and tests both rejected and supported token classes, whereas #47770 applies only a prefix guard.

    Related pull requests

    Duplicates

    #47718 and #47770 directly duplicate the #47708 Copilot ghp_* seed guard; #47718 is canonical because it is earlier, validator-based, and tested. Adjacent duplicate groups are #15920/#15944/#16101, #18256/#18298/#18755, #20602/#55528, #40961/#42475/#57137, and #65945/#66182/#66772 for their respective separate defects.

    Suggested consolidation

    Author action: rebase #47718 onto current main, or split out the part that can merge, preserving the validator-backed guard in agent/credential_pool.py and the ghp_*, github_pat_*, and gho_* regression cases. This keeps faith with the automated keep_open review on #47718, which confirms the unvalidated environment-ingestion path still exists, while recognizing that #47718 is the best existing fix but is not presently in the eligible lane; keep #47770 closed as its narrower duplicate and handle #20368's auxiliary resolver contract separately.

    Complex graph

    flowchart TD
        classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
        classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
        classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
        classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
        classDef best stroke-width:3px,stroke:#b45309
        classDef target stroke-width:3px,stroke:#4338ca
        I15914(["issue NousResearch/hermes-agent#15914 (closed)"])
        I15932(["issue NousResearch/hermes-agent#15932 (closed)"])
        I18254(["issue NousResearch/hermes-agent#18254 (closed)"])
        I21950(["issue NousResearch/hermes-agent#21950 (open)"])
        I47708(["issue NousResearch/hermes-agent#47708 (open)"])
        P19021["PR NousResearch/hermes-agent#19021 (closed)"]
        P20368["PR NousResearch/hermes-agent#20368 (open)"]
        P20602["PR NousResearch/hermes-agent#20602 (closed)"]
        P21964["PR NousResearch/hermes-agent#21964 (closed)"]
        subgraph Dup15920 ["PRs duplicating each other"]
            P15920["PR NousResearch/hermes-agent#15920 (closed)"]
            P15944["PR NousResearch/hermes-agent#15944 (closed)"]
            P16101["PR NousResearch/hermes-agent#16101 (merged)"]
        end
        subgraph Dup18256 ["PRs duplicating each other"]
            P18256["PR NousResearch/hermes-agent#18256 (closed)"]
            P18298["PR NousResearch/hermes-agent#18298 (closed)"]
            P18755["PR NousResearch/hermes-agent#18755 (merged)"]
        end
        subgraph Dup47718 ["PRs duplicating each other"]
            P47718["PR NousResearch/hermes-agent#47718 (open)"]
            P47770["PR NousResearch/hermes-agent#47770 (closed)"]
        end
        More["+7 more PRs not shown"]
        MoreI(["+3 more issues not shown"])
        P15920 -.->|unverified| I47708
        P15944 -.->|unverified| I47708
        P16101 -.->|unverified| I47708
        P18256 -.->|unverified| I47708
        P18298 -.->|unverified| I47708
        P18755 -.->|unverified| I47708
        P19021 -.->|unverified| I47708
        P20368 -.->|partial| I47708
        P20602 -.->|unverified| I47708
        P21964 -.->|unverified| I47708
        P47718 -->|best fix| I47708
        P47770 -->|fixes| I47708
        P15920 -->|fixes| I15914
        P15944 -->|fixes| I15914
        P16101 -->|best fix| I15914
        P15920 -->|fixes| I15932
        P15944 -.->|partial| I15932
        P16101 -->|best fix| I15932
        P18256 -.->|partial| I18254
        P18298 -->|fixes| I18254
        P18755 -->|best fix| I18254
        P21964 -->|best fix| I21950
        class I15914 closed
        class I15932 closed
        class I18254 closed
        class I21950 open
        class I47708 open
        class I47708 target
        class P15920 closed
        class P15944 closed
        class P16101 merged
        class P18256 closed
        class P18298 closed
        class P18755 merged
        class P19021 closed
        class P20368 open
        class P20602 closed
        class P21964 closed
        class P47718 open
        class P47770 closed
        class P16101 best
        class P16101 best
        class P18755 best
        class P21964 best
        class P47718 best
        class More closed
        class MoreI closed
        click I15914 "https://github.com/NousResearch/hermes-agent/issues/15914"
        click I15932 "https://github.com/NousResearch/hermes-agent/issues/15932"
        click I18254 "https://github.com/NousResearch/hermes-agent/issues/18254"
        click I21950 "https://github.com/NousResearch/hermes-agent/issues/21950"
        click I47708 "https://github.com/NousResearch/hermes-agent/issues/47708"
        click P15920 "https://github.com/NousResearch/hermes-agent/pull/15920"
        click P15944 "https://github.com/NousResearch/hermes-agent/pull/15944"
        click P16101 "https://github.com/NousResearch/hermes-agent/pull/16101"
        click P18256 "https://github.com/NousResearch/hermes-agent/pull/18256"
        click P18298 "https://github.com/NousResearch/hermes-agent/pull/18298"
        click P18755 "https://github.com/NousResearch/hermes-agent/pull/18755"
        click P19021 "https://github.com/NousResearch/hermes-agent/pull/19021"
        click P20368 "https://github.com/NousResearch/hermes-agent/pull/20368"
        click P20602 "https://github.com/NousResearch/hermes-agent/pull/20602"
        click P21964 "https://github.com/NousResearch/hermes-agent/pull/21964"
        click P47718 "https://github.com/NousResearch/hermes-agent/pull/47718"
        click P47770 "https://github.com/NousResearch/hermes-agent/pull/47770"
    
    Loading

    Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

    Cross-PR triage: Reviewed 19 pull requests and 8 issues in this complex. Each diff was read against this issue; Assessment working set: 108 kB of PR diffs, 55 kB of issue/PR text, 40 kB of discussion (58 comments), 37 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

  9. wang25669 commented on Aug 8, 2026

    @wang25669

    Real-world reproduction confirming this issue. Environment: Hermes Agent v0.20.0 (2026.8.3), Linux, install method git.

    Setup:

    • .env contains GITHUB_TOKEN=ghp_* (classic PAT, 40 chars, public_repo scope) — used for skills-hub / GitHub API rate limits
    • No COPILOT_GITHUB_TOKEN or GH_TOKEN set
    • Both copilot-provider and model-providers/copilot are in plugins.disabled

    Symptoms:

    1. /model picker still lists GitHub Copilot with a full model catalog (GPT-5.x, Claude, Gemini, etc.)
    2. Selecting any model fails immediately with HTTP 400: checking third-party user token: bad request: Personal Access Tokens are not supported for this endpoint
    3. hermes auth list shows copilot (1 credentials): GITHUB_TOKEN api_key env:GITHUB_TOKEN — the classic PAT was auto-seeded into the copilot credential pool even though the copilot plugin is disabled

    Root cause confirmed via source:

    • list_authenticated_providers() (hermes_cli/model_switch.py) never checks plugins.enabled — it only checks env var presence / credential pool / auth store
    • The credential pool auto-seeds from env:GITHUB_TOKEN because the copilot profile's env_vars includes GITHUB_TOKEN as a third-priority candidate
    • Per environment-variables.md, classic ghp_* PATs are explicitly "not supported" for the Copilot API — so this auto-derivation always produces a permanently-failing credential

    This is exactly the "auto-derive from classic ghp_* token" case described here. The picker shows a provider that can never work for free-GitHub-account users who only set GITHUB_TOKEN for skills-hub.

  10. sprd-tmu commented on Oct 4, 2026

    @sprd-tmu

    Corroboration: github_pat_* (fine-grained) fails identically — the admission gate is too narrow

    Second-order failure on current main (installed checkout ea81748579, Linux), Copilot Business plan. This extends the issue's scope: the suggested fix and the open PRs (#47718, #47770) only reject classic ghp_*, but fine-grained PATs are rejected by the chat endpoint too, and they pass validate_copilot_token() because github_pat_ is in _SUPPORTED_PREFIXES.

    Failure chain (each step verified live)

    1. COPILOT_GITHUB_TOKEN and GITHUB_TOKEN in ~/.hermes/.env held the same fine-grained PAT (github_pat_…, len 93) — identical secret_fingerprint on both credential_pool.copilot entries in auth.json, so pool priority was moot.
    2. validate_copilot_token() accepts it (github_pat_ is a supported prefix) → credential is enrolled.
    3. Exchange probe: GET api.github.com/copilot_internal/v2/token with Authorization: token github_pat_… → HTTP 404 (PATs cannot exchange).
    4. get_copilot_api_token() (hermes_cli/copilot_auth.py:489-499) silently falls back to the raw PAT; base_url stays on the generic https://api.githubcopilot.com instead of the account's real host.
    5. Every chat request → the issue's error verbatim:
      HTTP 400: checking third-party user token: bad request: Personal Access Tokens are not supported for this endpoint
      
      Identical against the plan-correct host (api.business.githubcopilot.com, discovered via GET /copilot_internal/user → copilot_plan: business, endpoints.api), so no host choice rescues a PAT.

    Why the current advice in the error string is misleading

    validate_copilot_token()'s rejection message tells users to use "A fine-grained PAT (github_pat_*) with Copilot Requests permission" — but step 5 shows the chat endpoint rejects PATs categorically ("checking third-party user token"), regardless of type or permissions. The only credential class that works end-to-end is an OAuth token that exchanges (device-flow ghu_ via the Iv1.b507a08c87ecfe98 client id → tid=… API token, HTTP 200, chat 200).

    Suggested acceptance-criteria additions

    • Treat all PAT prefixes (ghp_* and github_pat_*) as non-exchangeable for the chat path: either don't enroll them as copilot credentials, or enroll with a hard "exchange required" flag so the raw-token fallback in get_copilot_api_token() cannot send them to /chat/completions.
    • Fix the guidance string in validate_copilot_token() to stop recommending fine-grained PATs for Copilot API use.
    • (Matches this issue's existing ask) mark the credential unhealthy after this deterministic 400 so failover skips it.

    Resolution that worked here (no source patch)

    Device-code OAuth login with Hermes' shipped client id → ghu_ token in COPILOT_GITHUB_TOKEN, removed the pool entries, restarted (process env is loaded at startup and os.environ beats .env in resolve_copilot_token()). Exchange then returned 200 with endpoints.api=https://api.business.githubcopilot.com and chat completed 200. Side effect worth noting: while the exchange was failing, the model picker silently served the stale curated list (see NousResearch/hermes-labs-plugins#49) — the plan's real catalog (e.g. claude-sonnet-5.5) only appeared after auth was fixed, which made the broken state harder to diagnose.

    No credentials, org names, or hostnames included; token values reduced to prefix/length/fingerprint only.

  11. teknium1 commented on Oct 26, 2026

    @teknium1
    Collaborator

    Closing as fixed by the move. GitHub Copilot moved out of core into the github-copilot Hermes Labs plugin in #144788 (0591b66).

    Thanks @tcconnally for the detailed auth.json evidence.

    If you still see this with the github-copilot plugin installed (hermes plugins list), open an issue on NousResearch/hermes-labs-plugins.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existsarea/authAuthentication, OAuth, credential poolscomp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpointprovider/copilotGitHub Copilot (ACP + Chat)sweeper:risk-security-boundarySweeper risk: may affect sandboxing, auth, credentials, or sensitive datatype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions