Customizing SPIFFE ID format if using an external SPIFFE-compliant SDS should be supported

[bleggett]

Summary

1. When using the default Istio SDS, the current SPIFFE ID format should be the default
2. If using an alternative SPIFFE-compliant SDS, using an alternative SPIFFE ID format should be allowed without having to resort to DestRule hacks
3. Other parts of Istio should actively avoid introducing unnecessary implicit assumptions about the SPIFFE ID format and what is minting them to avoid creating more de-facto restrictions around their use, format, and the level of attestation a given user-supplied SPIFFE-compliant SDS server is actually configured for.

Detail

Currently, Istio uses a nonstandard variant of the SPIFFE ID spec, that mandates a SPIFFE ID format in the URI SAN field of the x509 workload certs:

spiffe://<trust_domain>/ns/<workload_namespace>/sa/<workload_service_account>

This means that workload certs minted by the default Istio SDS are indistinguishable - if I have 5 pods under the same service account, they share the same credentials, even if they may have different containers, run on different nodes, etc etc.

That is because the default Istio SDS is simplistic, does no granular workload identity attestation, and merely passes through trust and workload identity to K8S service accounts, which is Good Enough Most Of The Time.

Now that Istio supports replacing the default SDS provider with alternative SPIFFE-compliant SDS servers, such as SPIRE, this restriction makes less sense - the SDS server does (and should) control the format of the SPIFFE ID, and the granularity of the workload identity - for instance, if I use SPIRE with Istio and want to do workload attestation beyond just the service account level, I can easily do that today, and the SPIFFE ID format is defined with SPIRE, not Istio.

In fact, it is perfectly possible to do this today - I can integrate SPIRE with Istio as per our current docs, and configure SPIRE to mint SPIFFE IDs in a non-Istio-standard format, appending more granularity to the SPIFFE identifier to suit the level of attestation granularity my SPIFFE authority is actually engaging in:

spiffe://<trust_domain>/ns/<workload_namespace>/sa/<workload_service_account>/nodeid/<node_id/wl/<workload_name> - for instance

This works just fine with Istio, with the following exception - SPIFFE SAN validation is a hardcoded Envoy config that requires an exact match on spiffe://<trust_domain>/ns/<workload_namespace>/sa/<workload_service_account> - even though other forms of matching for SANs are supported by Envoy, we do not support them or expose them as configurable options.

This can be worked around with a DestinationRule such as the following:

# TODO destination rules need to be created for any SPIFFE IDs that don't follow the
# format that Istio expects (ns/NAMESPACE/sa/TARGET_POD_SVC_ACCOUNT)
# because ATM Istio defaults to clientside SAN checks that assume that SPIFFE ID format
# and this is not currently configurable
#
# Additionally, since DestinationRules override Istio's "default automTLS" settings, we need `mode: ISTIO_MUTUAL`
# in each DestRule to tell Istio that even though we have a custom destination config, we still want mTLS.
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: {{ .serviceName }}-custom-spire-destrule
spec:
  host: {{ .serviceName }}
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
      subjectAltNames:
      - spiffe://example.org/ns/{{ $.Release.Namespace }}/sa/{{ .serviceName }}/wl/foo

Once you do this, SPIFFE IDs can be constructed with whatever level of granularity you desire, and workload certs can be distinguishable by consumers at the level of attestation that is actually performed by the SDS, rather than the level of attestation that Istio's default SDS performs.

There has been resistance to changing the default SPIFFE ID format due to back compat with existing customer rules that also hardcode SPIFFE IDs in the format that the default Istio SDS emits, and that's reasonable - but given that we support pluggable SPIFFE-compliant SDS implementations there is no good reason why Istio itself should forbid or otherwise prevent customers from using an alternate SDS from using more granular SPIFFE IDs than the default.

Especially since this works fine today with a simple DestinationRule tweak, indicating that the problem is a simple set of currently-unconfigurable defaults, and not a systemic obstacle.

Frankly, outside of maybe requiring that a SPIFFE ID have at a minimum several expected parsable fields in it so Istio itself can extract the information it needs from SPIFFE IDs (/ns/<namespace> and /sa/<serviceaccount>/), it isn't really Istio's business what the SPIFFE ID format is - the SPIFFE ID format is and should be owned by the SPIRE-compliant SDS instance, and we support more than one SPIRE-compliant SDS instance. We just make bad assumptions elsewhere in the code that force those compliant instances to hew exclusively to the SPIFFE ID format our default SDS emits, which is an unnecessary restriction.

Affected product area (please put an X in all that apply)

[x] Ambient
[x] Docs
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[x] Security
[ ] Test and Release
[x] User Experience
[ ] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane

Additional context

[bleggett]

Some context:

#28712

#42114

[kyessenov]

Our authz API requires extracting the source principal from SPIFFE ID, for example, to restrict access by the source namespace. How would you support this API with custom SPIFFE IDs?

Similarly, our telemetry is very much "workload" oriented, meaning we drop the qualified pod name as soon as possible and only report the deployment name, in order to reduce the metric cardinality.

In general, using per-pod identity might give a false sense of security. Kubernetes itself doesn't distinguish between pods of the same KSA from authorization perspective (in RBAC, etc), and even then, tenancy by SA is very weak, and namespaces are much closer to an isolation unit.

[bleggett]

Our authz API requires extracting the source principal from SPIFFE ID, for example, to restrict access by the source namespace. How would you support this API with custom SPIFFE IDs?

Couple of ways I see

1. Allowing the AuthZ API to use SPIFFE ID URI matching to allow users to customize for themselves what constitutes "source principal" as defined by whatever SDS/workload CA they are using, if they want.

2. Or simply saying "whatever SPIFFE ID your workload CA mints, it MUST have /sa/<service_account> and /ns/<namespace> fields in it somewhere so K8S-service account authZ rules can work, but your workload CA/SDS server is free to add as many other fields to the SPIFFE ID as you need"

Similarly, our telemetry is very much "workload" oriented, meaning we drop the qualified pod name as soon as possible and only report the deployment name, in order to reduce the metric cardinality.

Do our telemetry APIs rely on parsing the ISTIO-SPIFFE ID format today? If they do, the above solutions would work. If they do not, they shouldn't be affected.

In general, using per-pod identity might give a false sense of security. Kubernetes itself doesn't distinguish between pods of the same KSA from authorization perspective (in RBAC, etc), and even then, tenancy by SA is very weak, and namespaces are much closer to an isolation unit.

Depends - my point is that attesting pod identity is the provenance and sole responsibility of the SDS server/workload CA you happen to be using - and the granularity to which identity is attested also belongs to that. Even today - the default istiod SDS/workload CA owns those guarantees, and is the trust root of all of them.

If we support pluggable SDS servers, and we do, we should consider respecting whatever the workload CA attests (or at least just respect the parts we care about and ignore the rest), rather than creating downstream de-facto assumptions that constrain what the workload CA can attest and encode in the cert, which is frankly backwards.

If you use the default istiod SDS workload CA, all we attest is Kubernetes service account, as attested by the Kubernetes API - but we are necessarily trusting SDS server to attest those things. If you swap that out for another SPIFFE-compliant SDS server, like say SPIRE, it can attest a superset of that - we don't have to care about the superset, but we shouldn't prevent the superset from being represented, which is what we do today.

[kyessenov]

1. Allowing the AuthZ API to use SPIFFE ID URI matching to allow users to customize for themselves what constitutes "source principal" as defined by whatever SDS/workload CA they are using, if they want.

Yes, but that's an API change. We're very wary of making any semantic changes to the existing APIs since any change can potentially break users.

2. Or simply saying "whatever SPIFFE ID your workload CA mints, it MUST have /sa/<service_account> and /ns/<namespace> fields in it somewhere so K8S-service account authZ rules can work, but your workload CA/SDS server is free to add as many other fields to the SPIFFE ID as you need"

That could work, but our implementation does strict regex matching I think. We'd need to make sure pattern matching is backwards compatible.

Do our telemetry APIs rely on parsing the ISTIO-SPIFFE ID format today? If they do, the above solutions would work. If they do not, they shouldn't be affected.

It matters because we report principals literally as primary metric tags. Having a pod name as a principal will overwhelm the metric systems (none of them scale well to POD^2 cardinality).

If you use the default istiod SDS workload CA, all we attest is Kubernetes service account, as attested by the Kubernetes API - but we are necessarily trusting SDS server to attest those things. If you swap that out for another SPIFFE-compliant SDS server, like say SPIRE, it can attest a superset of that - we don't have to care about the superset, but we shouldn't prevent the superset from being represented, which is what we do today.

If SPIFFE certs are only used by Istio, then it's better to propose to SPIRE to generate Istio-compatible identities, because Istio in general simply doesn't make use of pod names in the APIs. The only issue is inter-op with another system that shares the identities, and for that, we'd need more details on what the other system is.

[bleggett]

1. Allowing the AuthZ API to use SPIFFE ID URI matching to allow users to customize for themselves what constitutes "source principal" as defined by whatever SDS/workload CA they are using, if they want.

Yes, but that's an API change. We're very wary of making any semantic changes to the existing APIs since any change can potentially break users.

2. Or simply saying "whatever SPIFFE ID your workload CA mints, it MUST have /sa/<service_account> and /ns/<namespace> fields in it somewhere so K8S-service account authZ rules can work, but your workload CA/SDS server is free to add as many other fields to the SPIFFE ID as you need"

That could work, but our implementation does strict regex matching I think. We'd need to make sure pattern matching is backwards compatible.

Yep. Also, this would only matter to people using a nonstandard SDS. Anyone continuing to use the default istiod SDS shouldn't be affected - if you make an explicit choice to use a different SDS than the one we ship, we should support that and document it, but it (and the config required to support extended SPIFFE IDs) doesn't need to be the default.

Do our telemetry APIs rely on parsing the ISTIO-SPIFFE ID format today? If they do, the above solutions would work. If they do not, they shouldn't be affected.

It matters because we report principals literally as primary metric tags. Having a pod name as a principal will overwhelm the metric systems (none of them scale well to POD^2 cardinality).

That is useful info and something to consider.

If you use the default istiod SDS workload CA, all we attest is Kubernetes service account, as attested by the Kubernetes API - but we are necessarily trusting SDS server to attest those things. If you swap that out for another SPIFFE-compliant SDS server, like say SPIRE, it can attest a superset of that - we don't have to care about the superset, but we shouldn't prevent the superset from being represented, which is what we do today.

If SPIFFE certs are only used by Istio, then it's better to propose to SPIRE to generate Istio-compatible identities, because Istio in general simply doesn't make use of pod names in the APIs. The only issue is inter-op with another system that shares the identities, and for that, we'd need more details on what the other system is.

The only issue is inter-op with another system that shares the identities, and for that, we'd need more details on what the other system is. Or, we could not care about what other identity properties other systems want or need and simply act as a minimal intermediary between whatever workload CA you want to use that is compatible with us, and whatever external requirements you have - when it comes to SPIFFE, we are the ones that aren't compliant with (or allow fully-compliant implementations of) the published standard, and IMO that's on us, not SPIRE (or any other SDS).

We may require a subset of the spec for our own purposes, but we should not disallow (or refuse to pass thru) a superset of our requirements that are fully within the spec we support just due to some naive validation rules on our part - that's what we do today, and IMO that's a bug and demonstrably not strictly necessary if you are already using an alternate SDS/workload CA due to nonstandard requirements.

We should expect the things we need in the cert to be in the cert - if there are more things that external entities might want, that should be negotiated between the custom workload CA you are using to mint Istio workload certs, and your external entities - we shouldn't get in the middle of that and block it, or try to support all permutations of that ourselves.

[hzxuzhonghu]

For what case do you need more granular workload identity? For stateless application, k8s designed deployment as a logic concept for a group of instances, and each one has same permission. Why does istio need to separate them for auth?

[bleggett]

For what case do you need more granular workload identity? For stateless application, k8s designed deployment as a logic concept for a group of instances, and each one has same permission. Why does istio need to separate them for auth?

It doesn't - but there are external systems or integrations that will handle workload certs that might want or need that (see #42114 and @costinm's use cases), and Istio should not prevent you from putting more granular workload identity in the certs than what Istio itself needs. Especially since we support alternative workload CAs that provide more granularity - we just make it unnecessarily difficult to use that additional granularity.

Today Istio does prevent you from doing that, practically speaking, even if you use a non-default workload CA that supports this.

Additionally, we support replacing the istiod-default workload CA with alternate workload CAs which can attest a much more granular identity than istiod can - which is good - but rather than passing thru additional granularity that the customizable workload CA might put in workload certs, we put an upper bound on it

1. Istio itself doesn't need more granular identity attestation than the identity attestation that istiod's default workload CA supports.
2. Istio supports alternative workload CAs
3. Istio currently prevents you from using the more granular identity attestation of those external workload CAs because it insists on a de-facto standard, rather than simply insisting that the fields it needs are present, and ignoring+passing thru more granular specifiers.