ambient: rationalize and document data plane enablement

[howardjohn]

We need a table like https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/#controlling-the-injection-policy, but with ambient taken into account

Currently the logic is definitely "wrong", IMO, but we need to determine what is right. Then document it.

Related #43437

[costinm]

My take: the final state we want is ambient on by default in all the mesh, with a proper CRD defining interception policies, ideally vendor neutral and defined in GAMMA. K8S CNI doesn't provide an 'opt a pod out' - except maybe host network option.

Labels on namespace are used as an artifact of how mutating webhook works - this is our chance to move away from that.

I would think a user creating a Gateway with waypoint class in a namespace is a clear indication that workloads in that namespace need ztunnel (unless they have a sidecar). Can we do something similar for namespace using L4 only - for example define a Gateway class=ztunnel, which pilot ignores (doesn't create an actual deployment) but either is watched by CNI ( which will eventually need to start watching some CRDs for interception control ) or if it's easier just generate some internal-do-not-use-ztunnel label on namespace automatically ?

From user perspective, the config will be the same for L4 and L7 - creating a Gateway resource. Technically the spec does allow the implementation to actuate this by generating a common gateway - ztunnel is a valid multi-tenant L4 gateway
that happens to be deployed per node.

Later we could go a step further - if Istiod actuates gateways, and a 'class ztunnel' is created it could just deploy and manage ztunnel just like a gateway. The upgrade dance for ztunnel will likely be tricky and may be best managed by Istiod.

[howardjohn]

I think the biggest case for opting out is a sidecar (or proxy less), so there is some use case for opt out per pod IMO. Agree that may not be label driven. Another case is circular dependencies (API server and istiod), but that is only 1P deployments not arbitrary user ones probably.

Gateway for ztunnel enablement is definitely technically feasible. I neither love nor hate it right now, will think about it a bit more. It does seem
somewhat counter to the goal of "enabled everywhere"? Since you would enable it per namespace and there doesn't seem to be a way to say "class: no-ztunnel" ? Unless on-by-default means we auto generate that Gateway for them

[costinm]

Pining this to avoid closing. One comment: I think Istiod should be ambient-enabled - once we validate that CNI supports the port-exclude annotations. Port 15012 and 443 (injection) should be excluded from interception, 8080, 15010 and rest should be included. Same for proxyless - it should not opt-out the entire pod, but specific inbound and outbound ports. Proxyless uses are typically more advanced and not 'magically thinking' - using dedicated ports for gRPC is clean and simple, for cases where they want to do their own encryption. However it may be better to leave encryption to ztunnel and just handle LB and L7 policies - a much bigger problem is how to bypass the waypoint redirection ( or not bypass it if the policies can't be handled natively in proxyless ).

…

On Sat, Feb 25, 2023 at 10:19 AM John Howard ***@***.***> wrote: I think the biggest case for opting out is a sidecar (or proxy less), so there is some use case for opt out per pod IMO. Agree that may not be label driven. Another case is circular dependencies (API server and istiod), but that is only 1P deployments not arbitrary user ones probably. Gateway for ztunnel enablement is definitely technically feasible. I neither love nor hate it right now, will think about it a bit more. It does seem somewhat counter to the goal of "enabled everywhere"? Since you would enable it per namespace and there doesn't seem to be a way to say "class: no-ztunnel" ? Unless on-by-default means we auto generate that Gateway for them — Reply to this email directly, view it on GitHub <#43599 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUR2XPLBLMXTK7FG3ED5TWZJEK3ANCNFSM6AAAAAAVHKHZXM> . You are receiving this because you commented.Message ID: ***@***.***>

[andriilahuta]

Would it also be possible to keep the opt in behavior on a per-pod basis?
Sometimes it is useful to allow only a (small) subset of namespace's pods into the mesh. In such a case it could be quite inconvenient to opt out each pod across helm charts, operators, etc. And separating things into different namespaces is not always an option.

[costinm]

+1, per pod is pretty important for migrations as well. And for parity with current istio. However - if ambient moves into the CNI layer ( native implementations ) - it might ignore all options and be on by default in the entire cluster, and there is little we can do about it, just like today you can't turn off CNI encryption selectively. That's why I think for proxyless and other cases ( including Istiod port 15012 ) it may be useful to configure or detect ambient mode, and skip doing application-level mTLS. We'll have some migrations and mixed modes for some time - but I think the end goal should be ambient on by default with least config or options for users and all opt-outs disabled by admin.

…

On Sat, May 27, 2023 at 5:54 PM Andrii Lahuta ***@***.***> wrote: Would it also be possible to keep the opt in behavior on a per-pod basis? Sometimes it is useful to allow only a (small) subset of namespace's pods into the mesh. In such a case it could be quite inconvenient to opt out each pod across helm charts, operators, etc. And separating things into different namespaces is not always an option. — Reply to this email directly, view it on GitHub <#43599 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUR2RKUOYTLUMXMQEFAM3XIKO4TANCNFSM6AAAAAAVHKHZXM> . You are receiving this because you commented.Message ID: ***@***.***>

[keithmattix]

Not stale

[hanxiaop]

not stale