feat(auth): generic delegated access-token verifier (at+jwt) with external principal mapping

[jiashuoz]

Summary

Implements the generic delegated access-token verifier from the native-console surface design (§10.3–10.5): a disabled-by-default delegated: config block, token-kind dispatch that makes protected-header typ="at+jwt" delegated-owned before any existing credential path, an OIDC-discovery/JWKS verifier with the exact cache/refresh policy, the external_principal_mappings identity table with signed provisioning/attach population, a typed 401/503 wire split, and category-only metrics. A control plane (any OIDC issuer — configured entirely by deployment values, nothing operator-specific in source) can mint short-lived audience-bound tokens for its signed-in humans and call /v1 with account-scope authority mapped through the verified (iss, sub) pair.

Design conformance

• Dispatch (§10.4): authenticatePrincipal classifies by parse-only protected-header typ first, reading only typ and tolerant of every other header member (so an at+jwt with an oversized/oddly-typed kid/alg is still delegated-owned; the verification-time pins reject it later as a 401). A positively classified at+jwt never falls through — disabled, malformed, unavailable, or invalid — proven by tests that enroll the exact compact string as a valid API key and show it still rejects, including the odd-header variants. All other bearers keep today's precedence byte-for-byte (agent JWT → ate2a_ OAuth → API key → session cookie); the WS handshake is untouched (API-key only, by design).
• Verifier (§10.3): coreos/go-oidc discovery on a background retry loop (issuer network unavailability is never startup-fatal); one long-lived verifier per process. Post-signature pins: exact typ/alg-allowlist/single-string aud (arrays rejected even when they contain the right value)/iss/azp/singleton scope, exp − iat ≤ max_token_lifetime with skew only on iat-future/exp-past, bounded nonempty sub/jti, required context claims with allowed-value sets, forbidden claims rejected even as null, ≤32 top-level claims, bounded ASCII claim names, duplicate claims rejected. The required/forbidden claim policy is not defaulted in OSS (only protocol-level algorithm/lifetime/skew are) — enabling the verifier requires an explicit claim policy, so no operator-specific claim names or role values ship in OSS source.
• Exact limits: 16384-byte raw Authorization and compact token, 1024/8192/1024-byte decoded header/payload/signature, 32-byte typ, 16-byte alg, 128-byte kid, and every dual code-point/byte claim limit from the design table — each with limit / limit+1 boundary tests including multibyte and supplementary-plane cases.
• JWKS policy: 10 s fetch deadline, 65536-byte decoded response cap (JWKS and the OIDC discovery document), ≤32 keys, kid ≤128 ASCII bytes. The 600 s fresh window turns the cache over: a known kid is served directly only while fresh; past 600 s a rate-limited singleflight refresh is attempted and the cached kid is served only if that refresh fails, out to the +300 s stale grace (past 900 s even a known key is 503 until a refresh succeeds); a successful refresh with the kid gone is an immediate 401. Unknown kids are never served from stale state. Burst-1 token bucket refilling 6/60 s, 10 s negative-result cooldown, atomic replacement, failed refresh retains the last good set. No remote response bodies are logged, and refresh-outcome metrics are emitted outside the cache lock.
• Identity (§10.5): migration 108 external_principal_mappings(issuer, subject, user_id, created_at, PK (issuer, subject)). Auth performs one exact read-only (iss, sub) lookup → complete user (incl. account_class) → ScopeAccount with empty AgentID. Never subject-alone/email lookup, never trusts profile claims, never writes. users.google_subject keeps its exact legacy behavior.
• Population: provision gains optional external_issuer — when present it must byte-equal the configured delegated issuer (else 400 invalid_issuer, so a trailing-slash issuer can't produce an unauthenticatable mapping), and the provision transactionally inserts/replays the mapping; omitted preserves the pre-delegated code path (no mapping). Provision's external_ref now carries the §10.3 bound (1–128 code points / ≤512 bytes / no control chars, C0+DEL+C1) since it becomes the mapping subject — identical to the old 128-byte cap for ASCII refs. New signed POST /api/internal/users/external-principals/attach behind the same provisioning HMAC — 201 create / 200 same-triple replay / 409 external_principal_conflict / 404 user_not_found / 503 provisioning_not_configured | delegated_verifier_not_configured; body and issuer byte-compared against the one configured delegated issuer; never touches email/name/google_subject. Statuses/codes/success shape mirror the control plane's attach client exactly.
• 401/503 split: invalid anything (incl. unknown subject and verifier disabled) → existing 401 unauthorized + bare Bearer realm="e2a" challenge, no check-specific detail. Verifier-not-ready / JWKS transport with no usable cached key / cooldown / identity-store failure → 503 with the dedicated auth_unavailable availability code (Retryable) and no WWW-Authenticate, on both the legacy mux (text/plain) and the /v1 envelope. The credential is resolved exactly once per request (a per-request auth memo across the rate-limit middleware, the handler, and the challenge re-run) so the stateful JWKS bucket can't flip a 401 into a 503. Every existing credential keeps working through a total delegated outage.
• Metrics: e2a_delegated_auth_failures_total{category} (invalid_token, unknown_subject, verifier_unavailable, identity_store_failure) and e2a_delegated_jwks_refresh_total{outcome} (success, key_absent, transport_error, parse_error, rate_limited); bounded-enum labels, never subjects/issuer text/token data. Ops note: e2a-ops' metric-alert-coverage gate needs its allowlist entries landed before/with this merge.

Client surface checklist

• Go handler + integration tests
• Migration written + idempotent + safe on prod-sized tables (new empty table + index; no rewrites)

All other rows intentionally skipped: this adds no public /v1 surface and no client-visible API change. The attach endpoint is internal (like provision — deliberately absent from OpenAPI), delegated tokens are minted server-side by a control plane (never by SDK/CLI/MCP users), and openapi.yaml is untouched.

Operational risk

• Disabled by default; a deployment without a delegated: block is byte-for-byte unchanged except the dispatch classifier, which only diverts protected-header typ="at+jwt" bearers — a shape no existing e2a credential has (e2a agent JWTs carry typ as a payload claim).
• Enabled-but-misconfigured refuses to start (aggregated Validate errors; https issuer required in production). Issuer network outage degrades only delegated auth to 503; rollback = disable in place (enabled: false / E2A_DELEGATED_ENABLED=false) and redeploy.
• 503 availability responses use the dedicated auth_unavailable availability code (Retryable), regenerated cleanly through the catalog into the OpenAPI extension, the ErrorBody.Code doc vocabulary, docs/api.md, and the api/openapi.yaml golden — the only spec diff. This lets a code-first consumer tell an auth-backend outage from a genuine e2a bug and lets a 5xx-burn SLO except it.

Review response (house review round)

Fixed all three confirmed correctness bugs (each TDD'd — failing test first) plus the cleanups:

• M1 — classification applied verification pins. parseProtectedHeader (shared with Verify) enforced the typ/alg/kid byte caps + strict typing, so an at+jwt with an oversized/oddly-typed other header member was classified NOT-delegated and fell through to the API-key probe. Split a lenient peekProtectedTyp (reads typ only) for classification from the strict parse kept for verification. Proven at the classify seam and the agent dispatch layer.
• M2 — credential resolved 2–3× per request. The rate-limit middleware + handler + 401 challenge each re-ran Verify; the stateful JWKS burst bucket then flipped an unknown-kid 401 into a 503 and double-counted the metric. Added a one-shot per-request auth memo at the single authenticatePrincipal chokepoint (withRawRequest installs it; authChallenge reordered to run inside it). Proven at the agent layer (memo vs no-memo, stateful stub) and end-to-end through the real v1 chain (poll-limited op → 401, verifier run once).
• M3 — 600 s fresh window unimplemented. keyForKid served any cached kid for the whole 900 s, so a removed/compromised key kept verifying and §10.7's 725 s retention math was broken. Now fresh (≤600 s) serves directly; past 600 s a refresh is attempted and a known kid is served only on refresh failure, out to the stale grace. Tests assert a fetch at 601 s and that a rotated-out key stops verifying once a refresh succeeds.
• N1 provision now gates external_issuer by byte-equality (no unauthenticatable mappings). N2 provision's external_ref uses the §10.3 bound (noted above). N3 control-char rejection now spans C1 (0x80–0x9f). N6 JWKS metrics emitted outside the cache lock. N7 OIDC discovery body capped. N8 filled the §16 boundary gaps (dual code-point/byte + multibyte + supplementary-plane matrix, decodeSegment 1024/1025 signature edge, JWKS caps at limit, raw-auth at 16384); this surfaced and fixed boundedASCII admitting DEL. N9 documented the metrics + verifier/attach in observability.md/deployment.md. N4 WS-handshake comment added.
• N10 (my call): emptied the operator-specific claim defaults. The required/forbidden claim lists defaulted to the console's exact claim shape, which §10.3 calls deployment data and the public-boundary rule keeps out of OSS. Removed them (kept protocol-level algorithm/lifetime/skew); enabling now requires an explicit claim policy (validated), and config.example.yaml shows generic placeholders. The e2a-ops config supplies the real values. Ops note: the e2a-ops delegated: YAML MUST therefore include required_claims + forbidden_claims (no OSS fallback).

Test plan

• go build ./..., gofmt clean
• internal/delegated: full claim-rejection matrix, classification-tolerates-odd-header cases, boundary (limit/limit+1, multibyte, supplementary-plane) matrix + JWKS caps at limit + decodeSegment edges, JWKS fresh(600 s)/stale(900 s)/rotated-out/rate/cooldown/singleflight/malformed-response behavior and the discovery body cap, all with a fake clock + httptest issuer (RSA and EC), -race clean
• internal/agent: dispatch no-fallthrough (API-key-trap + odd-header proofs), resolve-once-per-request (memo vs no-memo, stateful stub) + end-to-end through the real v1 chain (401 not 503), mapped/unmapped/unavailable/store-failure outcomes with metric categories, wire-level 401-challenge/503-no-challenge, existing credential-precedence regressions (API key, agent JWT via the real bootstrap flow, tampered JWT, ate2a_, cookie fallback), attach lifecycle/HMAC/bounds/C1-control, provision with/without external_issuer and issuer-mismatch rejection, -race clean
• internal/httpapi: envelope-level 401/503 wire split (auth_unavailable); full package green including the error-vocabulary/literal-status/docs/api.md/OpenAPI-golden contract scans (golden regenerated — only diff is the new code)
• internal/identity + internal/config + internal/telemetry green
• Full go test ./...: green except 4 pre-existing failures (outreach/engagement counters: TestMarkSent*, TestInboundReviewApprovalAtomicallyAdvancesAuthenticatedEngagement, TestInboundActivityRecording_AuthenticatedMailUpdatesCounters) — reproduced identically on clean origin/main in a separate worktree, unrelated to this change
• After the control-plane side points at a local e2a with delegated.enabled: true: its local e2e lane green with the dev auth shim deleted

🤖 Generated with Claude Code

https://claude.ai/code/session_01NbqHH6fEZHS3JSGJpE5bqM