Handover 169 JWT

Handover — Finish #169 (Auth0 JWT signature verification)

Focused handover for taking Auth0 login JWT signature verification from its current shadow state to enforce, safely, and closing noetl/ai-meta#169.

Repo copies (source of truth): docs/HANDOVER-169-JWT.md (full reference) + docs/CODEX-169-HANDOFF-PROMPT.md (copy-paste Codex kickoff).

The one danger: prod runs a single server replica → no partial-replica canary. shadow verifies-and-meters but always allows. The moment you set enforce, a wrong issuer/audience/JWKS config rejects 100% of logins. Every enforce step is gated on a real success observation and paired with an instant one-variable rollback (NOETL_AUTH_VERIFY_SIGNATURE=shadow).

Where things stand (2026-07-04)

Leg	PR	State
Server sync-path RS256/JWKS verify	server#276	Merged → v3.52.0 → deployed to prod, running in shadow
Drive-path (Python-stdlib) mirror	e2e#84	OPEN / unmerged (same flag, default off; only runs when NOETL_AUTH_SYNC off)

Prod: server-rust:v3.52.0, single replica, NOETL_AUTH_VERIFY_SIGNATURE=shadow, NOETL_AUTH0_DOMAIN=mestumre-development.us.auth0.com (bare host), NOETL_AUTH0_AUDIENCE unset. JWKS reachable from the pod (2 RS256 keys). outcome="success" not yet captured — the verify metric is lazily registered and appears only after the first real claims-valid login hits shadow.

The finish sequence

1. Capture shadow success — trigger one real Muno login, read noetl_auth_jwt_verify_total{mode="shadow"}, require success only. Any bad_* / no_domain / jwks_unavailable = config mismatch to fix first.
2. Set audience — confirm the real aud (= SPA client_id, never printed) and set NOETL_AUTH0_AUDIENCE so enforce validates aud.
3. Merge e2e#84 — both paths verify (flag-gated; prod path is the sync one).
4. Flip to enforce — pre-checks pass → NOETL_AUTH_VERIFY_SIGNATURE=enforce → immediately verify a real login + latency → instant rollback to shadow at the first bad_*.
5. Close #169 once both paths verify, enforce is live + stable, and the metric shows success.

Flag tri-state (NOETL_AUTH_VERIFY_SIGNATURE)

• unset / off — claims-only decode (pre-#169, byte-identical).
• shadow — verify + meter + log, still allow. ← prod is here.
• enforce — verify and reject on failure. ← target.

The bare-host gotcha

verify_signature builds the issuer via format!("https://{domain}/"), so NOETL_AUTH0_DOMAIN must be the bare host — a URL value would produce https://https://…// and break every verify (a full login outage under enforce).

Exact prod Auth0 config to match

• Issuer: https://mestumre-development.us.auth0.com/
• JWKS: https://mestumre-development.us.auth0.com/.well-known/jwks.json
• Audience: the SPA client_id in GSM auth0_client.data.client_id (project noetl-demo-19700101) — value never printed.

Related

• Sessions Log · Releases · Codex Handover (broader Muno/travel + deploy handover)
• Issue: noetl/ai-meta#169