feat(providers): add gateway-owned credential refresh for short-lived provider tokens

[johntmyers]

Problem Statement

OpenShell needs first-class gateway-owned credential refresh for providers that use short-lived access tokens. The current provider model can inject static credentials into sandbox traffic, and recent provider work added provider profiles, attach/detach, custom profiles, and provider environment revision polling for running sandboxes. However, OpenShell does not yet own the lifecycle for refresh material such as OAuth refresh tokens, client secrets, or service-account private keys.

This blocks long-running sandboxes that depend on providers such as Microsoft Graph / Azure / Entra-backed APIs, Google Workspace, Google Vertex AI, IBM watsonx, ChatGPT/OpenAI OAuth-style integrations, and other enterprise SSO-bound APIs. Access tokens commonly expire during a sandbox session. Users should not have to restart sandboxes, expose refresh material inside the sandbox, or build one-off host refresh daemons.

Refresh material must stay in the OpenShell control plane / gateway. Sandboxes should continue to see only placeholder values or proxy-injected short-lived access tokens, and should observe refreshed access tokens through the existing provider environment revision path.

Proposed Design

Add gateway-owned provider credential refresh while keeping placeholder-based credential injection as the delivery path for this first chunk.

The core model should be:

• Store refresh material in a separate non-injectable refresh-state object type in the existing objects table.
• Keep Provider.credentials as the injectable current credential map used by GetSandboxProviderEnvironment.
• Add refresh metadata to provider profile credential declarations so profiles can describe how a credential is renewed.
• Add a gateway refresh worker that reads refresh-state objects, refreshes due credentials, writes the resulting short-lived access token into the owning provider's injectable credential key, and updates the provider object so the provider environment revision changes.
• Rely on the existing sandbox provider-env revision polling and ProviderCredentialState mechanics so running sandboxes observe refreshed credentials without restarting.
• Keep refresh tokens, client secrets, service-account JSON, private keys, and equivalent refresh material out of Provider.credentials and out of GetSandboxProviderEnvironment.

Initial refresh strategies should include:

• static: no active minting. Provider credential updates are picked up through the current provider environment revision path.
• external: OpenShell does not mint tokens. An external process updates the provider credential store, and running sandboxes pick up the update through the revision path.
• oauth2_refresh_token: gateway refreshes an access token using stored refresh-token material.
• oauth2_client_credentials: gateway mints access tokens using client credentials, covering Microsoft S2S / Entra-style flows.
• google_service_account_jwt: gateway signs a JWT assertion using stored service-account material and exchanges it for a Google access token, covering Vertex/GCP service-account use cases.

The first implementation should prioritize Microsoft S2S / Entra, Google OAuth refresh-token, and Google service-account / Vertex flows. The design should leave room for IBM watsonx IAM, AWS STS, OIDC variants, and other future refresh strategies without coupling them to the first implementation.

User-facing UX should include a way to inspect refresh status, force rotation, and update refresh configuration without exposing refresh material in normal provider output. Candidate CLI shape:

openshell provider refresh-status <provider>
openshell provider rotate <provider> [--credential <name>]
openshell provider refresh-config <provider> ...

Provider create/update should accept refresh material separately from injectable credentials, so users cannot accidentally place refresh tokens or service-account keys in the sandbox-visible credential map.

Related Requirements

• Enhanced Provider Management #896 tracks the overall provider v2 design, including declarative refresh strategies, max lifetime, lead time, fail-closed expiry, status/rotate UX, and refresh telemetry.
• feat(sandbox): supervisor should periodically refresh provider credentials for running sandboxes #1264 tracks the running-sandbox last-mile refresh requirement. Current code already has provider environment revisions and supervisor polling; this issue should use that path.
• Provider-managed OAuth credential refresh #1066 proposed provider-managed OAuth credential refresh with refresh material staying outside the sandbox.
• feat: credential refresh trait for short-lived provider tokens (Vertex AI first implementation) #472 proposed a gateway refresh trait for short-lived inference provider tokens, including Google Vertex, Azure OpenAI / Entra ID, and IBM watsonx.
• NemoClaw issues OpenAI/ChatGPT OAuth is not a viable NemoClaw provider path without manual host-side OpenClaw gateway bridging NemoClaw#1793, [Linux][Brev]Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset NemoClaw#2519, ollama proxy token diverges from stored token after re-onboard, causing persistent HTTP 401 on inference NemoClaw#2553, [Onboard] upgrade-sandboxes preflight requires stale credential env after provider switch from remote to local NemoClaw#2625, and [Ollama-local] Gateway routing proxy does not inject NEMOCLAW_OLLAMA_PROXY_TOKEN as Bearer on :11435 — silent hang on clean machine (continuation of #2519) NemoClaw#3198 show adjacent credential-state problems: stale cached credentials, local/no-auth providers being treated as remote auth, and runtime injection mismatches. This OpenShell design should explicitly model no-auth, static, external-refresh, and gateway-refreshing credentials so stale or non-applicable credentials do not keep affecting providers.

Alternatives Considered

• Store refresh tokens directly in Provider.credentials. Rejected because Provider.credentials is the injectable map returned through GetSandboxProviderEnvironment; refresh material must not enter the sandbox.
• Wait for profile-side/proxy-side credential injection before refresh. Rejected because placeholder-based injection can already deliver refreshed access tokens, and short-lived token demand is immediate.
• Require host-side refresh daemons. Rejected because this duplicates provider-specific logic, fails for long-running autonomous sandboxes, and moves lifecycle ownership outside OpenShell.
• Add a new database table for refresh state. Rejected for this scope; profile and provider follow-up work has kept provider storage in the existing objects table.
• Implement a single provider-specific refresher first. Rejected because Microsoft S2S, Google OAuth, and Google service-account flows are already known requirements and need a shared scheduler/state model.

Agent Investigation

The current repo already has much of the sandbox-side propagation path needed for refreshed credentials:

• Provider.credentials is the current secret map on provider records.
• GetSandboxProviderEnvironment resolves attached providers into an environment map and returns provider_env_revision.
• crates/openshell-sandbox/src/provider_credentials.rs maintains revision-scoped credential snapshots and keeps recent generations so existing placeholders continue to resolve.
• The sandbox settings poll loop refreshes provider environment when provider_env_revision changes and installs the new environment into ProviderCredentialState.
• The proxy resolves placeholders per request through SecretResolver, so updated provider credential values can affect later proxied requests without mutating already-running process environments.
• Provider profiles currently define credentials, endpoints, binaries, category, and inference capability, but they do not yet define refresh metadata.
• Custom provider profiles are stored in the existing objects table; this issue should use the same persistence boundary for refresh-state objects rather than adding new tables.
• Existing OpenShell CLI OIDC login code already contains OAuth refresh-token helper logic for gateway authentication, but provider refresh needs separate gateway-side storage, scheduling, status, and provider credential updates.

Non-Goals

• Do not replace placeholder-based credential injection in this issue.
• Do not implement profile-side/proxy-side credential injection in this issue.
• Do not expose refresh tokens, client secrets, service-account JSON, or private keys through provider env resolution.
• Do not add new database tables.
• Do not make already-running process environments mutable.
• Do not implement every possible provider strategy in the first PR; keep the strategy interface extensible.

[johntmyers]

🏗️ build-plan

Implementation Plan

Issue type: feat
Complexity: High
Confidence: Medium — the storage and refresh path are clear, but the external strategy contract and final refresh-metadata placement need one design choice up front.

Summary

Add gateway-owned refresh for short-lived provider tokens by keeping Provider.credentials as the current injectable access-token map and moving refresh material into a separate non-injectable object stored in the existing objects table. The gateway refresh worker should update only the provider record on successful refresh so the existing GetSandboxProviderEnvironment revision flow, sandbox ProviderCredentialState, and inference route resolution continue to work without sandbox-side redesign.

Codebase Mapping

• proto/datamodel.proto, proto/openshell.proto: add the new stored provider-refresh object, refresh status/config RPCs, and provider profile refresh metadata.
• crates/openshell-core/src/metadata.rs: register object traits for the new stored refresh object.
• crates/openshell-providers/src/profiles.rs, providers/*.yaml: extend profile schema and validation with refresh metadata; seed Microsoft and Google defaults first.
• crates/openshell-server/src/grpc/provider.rs, crates/openshell-server/src/grpc/mod.rs, new crates/openshell-server/src/provider_refresh.rs, crates/openshell-server/src/lib.rs: persist refresh material/status, expose refresh UX RPCs, run the scheduler, and write refreshed access tokens back into Provider.credentials.
• crates/openshell-server/src/grpc/policy.rs, crates/openshell-server/src/inference.rs: reuse existing provider-env and inference resolution; add regression coverage proving provider updates trigger new revisions and refreshed inference credentials.
• crates/openshell-cli/src/main.rs, crates/openshell-cli/src/run.rs, crates/openshell-cli/tests/*: add refresh-status, rotate, refresh-config, and keep provider create/update focused on injectable credentials only.

Implementation Steps

1. Define the data model and API surface: add a dedicated non-injectable provider-refresh object in the existing objects table, add refresh strategy metadata to provider profiles, and add minimal RPCs for refresh-status, refresh-config, and rotate.
2. Implement gateway persistence and validation: store refresh material and status separately from Provider.credentials, preserve existing provider read contracts, and keep CreateProvider/UpdateProvider scoped to injectable current credentials only.
3. Add the refresh engine: introduce a background gateway worker that scans configured refresh objects, schedules refresh before expiry, serializes refresh per provider, and implements static, external, oauth2_refresh_token, oauth2_client_credentials, and google_service_account_jwt, with full first-pass coverage for Microsoft S2S/Entra, Google OAuth refresh-token, and Google service-account/Vertex.
4. Reuse the existing credential propagation path: on successful refresh, update only Provider.credentials so existing provider env revision polling, sandbox placeholder rotation, and inference-route resolution observe new credentials without mutating running process envs.
5. Wire CLI UX: add openshell provider refresh-status, openshell provider refresh-config, and openshell provider rotate, and move refresh-material input out of normal provider create/update flows.
6. Finish with targeted profile and docs updates: add refresh metadata to prioritized profiles first, then document the new refresh lifecycle and user-facing commands.

Test Plan

• Unit tests: profile serde/validation for refresh metadata; new object metadata/accessors; strategy-specific refresh helpers; scheduler timing/backoff/state transitions; redaction tests proving refresh material never appears in provider read APIs.
• Integration tests: gateway provider tests for create/update vs refresh-config separation, successful refresh updating Provider.credentials, failed refresh preserving last-known-good credentials, compute_provider_env_revision changing when the provider record changes, and inference route resolution seeing rotated provider tokens.
• CLI integration tests: parsing and execution of new provider refresh commands, plus clear error cases for invalid refresh-config/material input.
• E2E tests: one running sandbox attached to a refresh-managed provider, backed by a fake token issuer, proving the gateway refresh updates the provider record and the sandbox observes the new provider_env_revision through the existing poll loop; add inference-route coverage if the refreshed provider is used for managed routes.

Docs Impact

• Update architecture/gateway.md with the provider refresh ownership boundary and background worker lifecycle.
• Update docs/sandboxes/manage-providers.mdx for refresh configuration, status, rotation, and supported strategies.
• Update docs/sandboxes/inference-routing.mdx if refreshed provider credentials become part of the supported inference path.

Security Risks & Open Questions

• CWE-200: refresh tokens, client secrets, and service-account keys must stay non-injectable, redacted from all read APIs, and excluded from logs and OCSF messages.
• CWE-918: if token endpoints or issuer URLs are configurable, validate them against profile metadata or an explicit allowlist so the gateway refresh worker cannot become an SSRF primitive.
• CWE-362: refresh must be single-writer per provider; failed or concurrent refreshes must not clobber the current credential map or cause unnecessary revision churn.
• Open question: define the exact external strategy contract before implementation; the issue names the strategy but not its fetch/request model.
• Open question: confirm whether refresh metadata should live on ProviderProfileCredential or ProviderProfile; credential-level metadata is the lower-risk fit because env-var injection is already modeled there.

---

Revision 1 — initial plan