[FEATURE] Workload identity for AI agents via SPIFFE/SPIRE

[inference-gateway-maintainer]

Summary

Give every workload this operator manages - Gateway, Agent, MCP, and Orchestrator - a cryptographically verifiable, attested, short-lived, automatically rotated workload identity based on SPIFFE (Secure Production Identity Framework For Everyone) and its reference implementation SPIRE (a CNCF graduated project).

SPIFFE is the de-facto open standard for "what is this workload, cryptographically." It is the right foundation for AI-agent identity: agents are non-human identities that call other agents (A2A), tools (MCP servers), and downstream model providers, often on behalf of a user. Today those calls are secured with static, long-lived secrets and unattested TLS. A SPIFFE identity (an SVID) lets each workload prove who it is to its peers with no pre-shared secret, enabling zero-trust mTLS between our components and, later, keyless access to external systems.

This is proposed as an opt-in, feature-gated capability (the SPIRE control plane is an optional cluster dependency), framed in phases - mirroring the approach taken in #112 for agent-sandbox.

Background: the AI-agent identity problem

An AI-agent platform has a fan-out of machine-to-machine calls that all need to answer "who is calling, and on whose behalf":

• Orchestrator -> discovered Agent and MCP CRs (A2A + MCP traffic)
• Gateway -> discovered MCP CRs and upstream model providers
• Agent -> Agent (agent-to-agent / A2A) and Agent -> MCP

The industry is converging on a two-layer model for this:

1. Workload identity (this issue): a verifiable identity for the running workload itself, anchored in platform attestation rather than a secret. SPIFFE/SPIRE is the standard here.
2. Delegation / authorization (complementary, see below): short-lived, scoped tokens that carry "on behalf of whom, with what permissions" across the call chain - OAuth 2.1 token exchange, IETF transaction tokens, the MCP authorization spec, and A2A agent-card auth schemes. SPIFFE provides the trust anchor these layers bind to.

We already have pieces of layer 2 (OIDC client auth on the Gateway). We have nothing for layer 1.

Current state (the gap)

Identity in the operator today is static and unattested:

• Kubernetes ServiceAccounts. reconcileRBAC / reconcileServiceAccount (internal/controller/gateway_controller.go) create a ServiceAccount per Gateway (ServiceAccountSpec in api/v1alpha1/gateway_types.go). That is a bearer-token identity scoped to the Kubernetes API - not an identity our workloads can present to each other or to external services, and not cryptographically attested at the workload level.
• TLS is statically provisioned. TLSConfig / TLSSpec reference a Secret with a cert and key (api/v1alpha1/gateway_types.go, agent_types.go); RoutingTLSSpec can wire up a cert-manager ClusterIssuer for the listener. These secure the transport but are not tied to a workload identity, are not mutually authenticated by default, and rotate on cert-manager's schedule rather than per-workload on a short TTL.
• Auth is client-facing OIDC only. AuthSpec / OIDCSpec authenticate clients to the Gateway. There is no mutual authentication between Gateway, Agent, MCP, and Orchestrator pods.
• Cross-CR traffic is unauthenticated at the workload layer. The Gateway reaches MCPs via the MCP_SERVERS env var; the Orchestrator mounts ~/.infer/agents.yaml and ~/.infer/mcp.yaml. These carry URLs, not identities - any pod that can reach the address can talk to it.
• Long-lived static credentials to external systems. Provider API keys (ProviderSpec, plus the secrets RBAC the Gateway holds), the Orchestrator's Telegram token (TokenSecretRef) and LLM APIKeySecretRef are long-lived secrets mounted into pods. There is no path to short-lived, identity-derived credentials.

A grep confirms no controller references SPIFFE, an SVID, or a workload-identity socket.

What SPIFFE/SPIRE provides

• SPIFFE ID - a URI naming the workload: spiffe://<trust-domain>/<path>, e.g. spiffe://inference-gateway.com/ns/<namespace>/sa/<service-account>.
• SVID (SPIFFE Verifiable Identity Document) - the credential proving the SPIFFE ID, as an X.509-SVID (for mTLS) or a JWT-SVID (for app-level auth and OIDC federation). SVIDs are short-lived (X.509-SVIDs default to ~1h) and auto-rotated.
• Workload API - a local unix-domain-socket API the workload calls to fetch its SVID and the trust bundle. The workload bootstraps with no secret; SPIRE decides what identity to hand back based on attestation.
• Attestation - the SPIRE Agent verifies node identity (on Kubernetes via the PSAT - Projected Service Account Token - node attestor) and workload identity (k8s workload attestor selectors: k8s:ns:<ns>, k8s:sa:<sa>, k8s:pod-label:<k>:<v>, k8s:container-image:<img>). This binds the SVID to the actual pod, not to a copied secret.
• Federation - trust bundles can be exchanged across trust domains, so an Agent in one cluster/org can verify an Agent in another. This is the cross-org A2A story.

The Kubernetes integration we would build on

• SPIRE Helm charts (spiffe/spire + spiffe/spire-crds) deploy SPIRE Server + per-node Agent.
• SPIRE Controller Manager provides CRDs that automate registration: ClusterSPIFFEID (declaratively map pods -> SPIFFE IDs via label/namespace selectors), ClusterFederatedTrustDomain, and ClusterStaticEntry (group spire.spiffe.io).
• SPIFFE CSI Driver (csi.spiffe.io) mounts the Workload API socket into a pod via a read-only ephemeral inline volume - no hostPath.

Because every workload we emit is already labelled app: <name> (see agent_controller.go, gateway_controller.go), a ClusterSPIFFEID podSelector maps cleanly onto our existing pods.

Proposed approach (phased, opt-in)

Phase 0 - Identity plumbing in the operator (no hard dependency)

Add a shared, opt-in IdentitySpec to all four CRDs (alongside the existing shared ServiceAccountSpec / TLSSpec), default disabled:

// api/v1alpha1 (shared type, embedded in Gateway/Agent/MCP/Orchestrator specs)
type IdentitySpec struct {
    // Enabled turns on SPIFFE workload identity for this workload.
    Enabled bool `json:"enabled,omitempty"`
    // Mode selects the identity backend. Only "spiffe" is supported today.
    // +kubebuilder:default=spiffe
    Mode string `json:"mode,omitempty"`
    // TrustDomain optionally overrides the cluster default trust domain.
    TrustDomain string `json:"trustDomain,omitempty"`
    // CSIDriver mounts the Workload API socket (default "csi.spiffe.io").
    CSIDriver string `json:"csiDriver,omitempty"`
    // SocketPath is the Workload API endpoint exposed to the container.
    // +kubebuilder:default="unix:///spiffe-workload-api/spire-agent.sock"
    SocketPath string `json:"socketPath,omitempty"`
    // ManageClusterSPIFFEID lets the operator create a ClusterSPIFFEID for
    // this workload. Requires the SPIRE controller-manager CRDs (feature-gated).
    ManageClusterSPIFFEID *bool `json:"manageClusterSPIFFEID,omitempty"`
    // FederatesWith lists trust domains to federate with (rendered into the
    // ClusterSPIFFEID).
    FederatesWith []string `json:"federatesWith,omitempty"`
}

When identity.enabled: true, the reconciler:

1. Mounts the Workload API socket into the workload pod via the SPIFFE CSI Driver and sets the standard env var so the runtime can find it:

   volumes:
     - name: spiffe-workload-api
       csi: { driver: csi.spiffe.io, readOnly: true }
   # container
   volumeMounts:
     - { name: spiffe-workload-api, mountPath: /spiffe-workload-api, readOnly: true }
   env:
     - { name: SPIFFE_ENDPOINT_SOCKET, value: "unix:///spiffe-workload-api/spire-agent.sock" }

2. Optionally creates a ClusterSPIFFEID for the workload (when manageClusterSPIFFEID is true and the CRD is present), selecting the pods by the app: <name> label we already set:

   apiVersion: spire.spiffe.io/v1alpha1
   kind: ClusterSPIFFEID
   metadata: { name: <name> }
   spec:
     spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
     podSelector: { matchLabels: { app: <name> } }
     # federatesWith: [...]   # from spec.identity.federatesWith

3. Feature-gates the CRD dependency exactly like [FEATURE] Stronger Isolation via SecurityContext + optional agent-sandbox backend #112 does for agent-sandbox: detect spire.spiffe.io via RESTMapper/discovery; if manageClusterSPIFFEID is requested but the CRD is absent, set Ready=False with a clear reason instead of crashing. Register the type in the scheme and Owns(&ClusterSPIFFEID{}) only when present.

4. Surfaces results on status: add status.spiffeID and a WorkloadIdentityReady condition (plus a printer column on each kind).

5. Adds the kubebuilder RBAC markers for spire.spiffe.io (clusterspiffeids, and likely clusterfederatedtrustdomains) and regenerates config/rbac + manifests/*.yaml.

Phase 0 ships the identity to the pod and registers it; it does not yet change how the workloads talk to each other (that needs the runtime images).

Phase 1 - SVID-based mTLS in the data path + federation (cross-repo)

Have the runtime images consume the Workload API to:

• Establish mTLS on the cross-component hops using X.509-SVIDs: Gateway <-> MCP, Orchestrator <-> Agent/MCP, Agent <-> Agent (A2A). Each side authorizes the peer's SPIFFE ID.
• Use ClusterFederatedTrustDomain + spec.identity.federatesWith for cross-cluster / cross-org agent communication.

The operator's role here is to provide the socket + trust bundle and the config toggles; the TLS handshake logic lives in the gateway / agent / MCP / CLI images (inference-gateway/*). Track the runtime-side work as cross-repo issues.

Phase 2 - Keyless credentials and delegation (cross-repo, longer term)

Replace long-lived static secrets with short-lived, identity-derived credentials:

• Exchange a JWT-SVID for cloud IAM credentials or model-provider tokens via OAuth 2.0 Token Exchange (RFC 8693) and/or SPIRE's OIDC Discovery Provider (JWKS), removing static provider API keys (ProviderSpec) and the Orchestrator's APIKeySecretRef where the provider supports federation.
• Bind the delegation / authorization layer to the SPIFFE anchor (see next section): MCP authorization tokens, A2A agent-card auth, and transaction tokens that carry user-delegated, scoped context across the agent call chain.

Complementary standards (the broader landscape)

SPIFFE is the workload-identity anchor; these are the "on whose behalf, with what scope" layers that build on it. Worth tracking so the design stays compatible:

• IETF WIMSE (Workload Identity in Multi-System Environments) WG - standardizing workload identity tokens and token exchange across systems, explicitly targeting agentic call chains.
• OAuth 2.0 Token Exchange (RFC 8693) - delegation / impersonation; the bridge from an SVID to a downstream-scoped token.
• IETF Transaction Tokens (draft-ietf-oauth-transaction-tokens) - short-lived tokens that carry call-chain context across services/agents.
• MCP authorization spec - MCP servers are OAuth 2.1 resource servers; per the current spec (2025-11-25) clients MUST implement OAuth 2.0 Protected Resource Metadata (RFC 9728) for AS discovery and Resource Indicators (RFC 8707), with optional Dynamic Client Registration (RFC 7591). Directly relevant to the MCP kind.
• A2A protocol (Linux Foundation) - agents advertise auth via the Agent Card using standard HTTP security schemes; SPIFFE/mTLS is a natural transport binding. Relevant to the Agent kind.
• OpenID Foundation - AI agent identity work, including the Identity Assertion Authorization Grant ("cross-app access") for enterprise SaaS.

Recommendation

Ship Phase 0 as an opt-in spec.identity capability (cheap, self-contained, feature-gated on the optional SPIRE CRDs - no behavior change when disabled). Adopt Phase 1 mTLS once the runtime images consume the Workload API. Track Phase 2 (keyless credentials + delegation) as a longer-term cross-repo initiative. This sequencing matches the phased, opt-in pattern established in #112.

Cross-repo impact

• operator: shared IdentitySpec + status fields, reconciler plumbing (CSI volume, env, optional ClusterSPIFFEID), feature-gating via discovery, RBAC markers, regenerated manifests/*.yaml, samples, e2e (with SPIRE installed in the k3d-dev cluster).
• inference-gateway/docs: [DOCS] ticket for the new fields and a "Workload identity (SPIFFE/SPIRE)" guide.
• gateway / agent / MCP / CLI images: Phase 1 (consume the Workload API, do SVID mTLS) and Phase 2 (token exchange / delegation).
• Helm / install manifests: ship the new RBAC; document the optional SPIRE dependency.

References

• SPIFFE overview and concepts: https://spiffe.io/docs/latest/spiffe-about/overview/
• SPIRE: https://github.com/spiffe/spire - SPIRE Controller Manager (ClusterSPIFFEID): https://github.com/spiffe/spire-controller-manager
• SPIFFE CSI Driver: https://github.com/spiffe/spiffe-csi - Helm charts: https://github.com/spiffe/helm-charts-hardened
• IETF WIMSE WG: https://datatracker.ietf.org/wg/wimse/about/
• OAuth 2.0 Token Exchange (RFC 8693): https://www.rfc-editor.org/rfc/rfc8693 - Transaction Tokens: https://datatracker.ietf.org/doc/draft-ietf-oauth-transaction-tokens/
• MCP authorization: https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization - A2A: https://a2a-protocol.org
• Related operator issue: [FEATURE] Stronger Isolation via SecurityContext + optional agent-sandbox backend #112 (isolation via SecurityContext + optional agent-sandbox backend)

Acceptance Criteria

• Shared IdentitySpec (enabled, mode, trustDomain, csiDriver, socketPath, manageClusterSPIFFEID, federatesWith) added and embedded in Gateway, Agent, MCP, and Orchestrator specs; default disabled (no behavior change when off)
• When identity.enabled: true, the workload pod mounts the SPIFFE CSI Driver volume and gets SPIFFE_ENDPOINT_SOCKET
• When manageClusterSPIFFEID is true and the spire.spiffe.io CRDs are present, the operator creates/updates a ClusterSPIFFEID selecting the workload's app: <name> pods (and renders federatesWith)
• CRD dependency is feature-gated via RESTMapper/discovery: if requested but absent, set Ready=False with a clear reason instead of crashing; scheme registration and Owns(&ClusterSPIFFEID{}) happen only when present
• status.spiffeID and a WorkloadIdentityReady condition are surfaced, with a printer column per kind
• kubebuilder RBAC markers for spire.spiffe.io added; config/rbac and manifests/*.yaml regenerated and committed
• Unit + envtest coverage for the enabled/disabled paths and the CRD-absent feature-gate path
• e2e installs SPIRE in k3d-dev and asserts a workload receives an SVID for its expected SPIFFE ID
• Samples under config/samples / examples demonstrate an identity-enabled workload
• Phase 1 (SVID mTLS in the data path) and Phase 2 (keyless credentials / delegation) filed as cross-repo follow-ups; docs [DOCS] ticket opened