[Design Proposal] OTel exporter (tracing) token management

[AnoshanJ]

Problem

When an agent is deployed, or an external agent is connected, the platform issues a tracing API key: an RS256 JWT signed by agent-manager-service, sent by the instrumentation SDK on every trace export in the x-amp-api-key header. Today:

• Validity is hardcoded to 1 year (generateAgentAPIKey in services/agent_manager.go).
• There is no way to revoke or rotate a token. A leaked token can publish traces into an organization's tenancy until it expires.
• The control plane does not record token expiry anywhere, so nothing knows when a deployed agent's token will stop working.
• Related UX bug (Do not systematically recreate AMP_KEY when "Setup Agent" is clicked #1140): every click of "Setup Agent" on an external agent silently mints a new token

Ref: #1063.

Existing Solutions

• Observability platforms such as Datadog and Grafana Cloud use opaque ingest keys that can be revoked server side, since validation is stateful.
• This repo already has full create/revoke/rotate for agent invocation API keys (services/agent_apikey_service.go), with changes pushed to gateways over websocket plus a bulk-sync fallback (services/apikey_broadcaster.go). Tracing keys have none of this because they are stateless JWTs that the gateway validates against JWKS only.
• Current workaround: redeploying an internal agent regenerates its token. External-agent users can generate a new token from the console; old tokens stay valid until expiry.

Proposed Solution

Overview

No single option solves this, so the options are ranked.

Option 1: short default expiry plus manual regenerate with custom expiry

Phase 1): Make the deployed-agent token expiry configurable instead of the hardcoded 1 year duration, and add a regenerate flow:

• New endpoint POST /orgs/{org}/projects/{proj}/agents/{agent}/tracing-token/regenerate It mints a new token with optional expiry parameter, upserts the KV secret (the secret reference is stable, so traits don't change), and restarts the workload by stamping restartedAt on the ReleaseBinding. That is a pod rollout, not a full redeploy; clients/openchoreosvc/client/deployments.go already does this internally.
• Console: a refresh action next to the Auto-Instrumentation toggle, with an expiry selector and a confirmation dialog, since the agent restarts.
• New expiry-tracking columns on internal database, so the console and the phase 2 job know when tokens expire.

Challenges: without auto-rotation, a short default risks trace outages when tokens expire unnoticed. Phase 1 therefore keeps a conservative default (90 days, open to feedback) and documents the manual refresh responsibility. Old tokens stay valid until their expiry.

Optional Phase 2: auto-rotation job

A background scheduler in agent-manager-service, following the existing monitor scheduler pattern. It finds tokens expiring within a threshold (around 7 days), regenerates them, upserts the secret, and restarts the workload. With this in place a 30-day default becomes safe.

Option 2: persist jti and expiry, revoke with gateway support

Add a jti claim to issued tokens, persist issued-token records (jti, agent, environment, expiry), and expose revoke APIs and UI. Revoked jtis would reach gateways over a sync channel with the gateway. Denylist entries expire with the token, so state stays bounded.

Blocker: the gateway's jwt-auth policy validates statelessly and has no denylist support, so this needs an enhancement on the API Platform gateway side. The jti claim itself can ship in phase 1, so tokens issued from now on become revocable once the gateway supports it.

Additionally: document signing-key rotation (already possible)

The JWKS already supports multiple keys with an active-key selector (JWT_SIGNING_ACTIVE_KEY_ID). Rotating the RSA key and dropping the old kid kills every outstanding tracing token within the gateway's JWKS cache TTL (about 5 minutes). This needs only be documented.

Option 5: Thunder AgentID tokens (not pursued)

Per-agent Thunder client_credentials identities already exist and could in theory be exchanged for short-lived tracing tokens with custom claims. But AgentID exists for agent access control, not trace publishing. Reusing it couples two unrelated credential lifecycles and requires token-refresh logic in both instrumentation SDKs. Listed only for completeness.

Out of Scope

• Gateway policy changes
• Switching trace ingest to opaque keys: the OTEL route depends on JWT claim-to-header mapping for tenant attribution, which the opaque-key policy cannot provide.
• SDK-side token reload without pod restart.

MCP, CLI, and Skill Components

Component	Required (yes/no)	Reason
MCP	No	The external-agent MCP tool already returns a token at creation. A regenerate tool can come later if agent workflows need it.
CLI	No	amctl already outputs a token on external-agent create. A rotate command is a possible follow-up once the REST endpoint exists.
Skill	No	Token lifecycle management has no agent-facing skill surface.

Open Questions

• Should we rename the 'x-amp-api-key' to 'x-am-api-key'

Milestones

Phase	Scope	Target
1	Fix #1140, configurable expiry, regenerate endpoint + console UI, expiry tracking columns, jti claim, docs including the signing-key-rotation runbook	TBD
2	Auto-rotation job, drop default expiry to 30 days	TBD
3	jti-based revocation (revoke API, UI, gateway sync)	Blocked on gateway

Dependencies: Phase 3 requires denylist support in the API Platform gateway's jwt-auth policy.