Support injecting init container to pods backed by ztunnel

[yuval-k]

The init container appraoch to solve the CNI race problem. This requires a mutating web hook to inject an init container. The init container will block the application pod from starting, until the ztunnel sockets are present.

Notes:

• To make the distinction between sidecar and ambient pod we pass in "/redirection/ambient" as part of the webhook http path.
• This does mean that the ambient pod selection logic is "duplicated" in the webhook config.
• I'll wait on Untaint controller #48818 to merge first, as i'll want to re-use some the code from there for integration testing.

[istio-testing]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[bleggett]

Suggested change

	ZtunnelTemplateName = "ztunnel"
	ZtunnelTemplateName = "ztunnel"

nit: do we need to call it ztunnel? it strictly speaking doesn't have anything to do with ztunnel, right?

ambient or cni-wait or ambient-init?

[bleggett]

EDIT: Nevermind. I see, it is waiting for the ztunnel sockets rather than direct CNI readiness.

I still think the name is overly inside-baseball tho, in general I don't wanna proliferate ztunnel in our explicit or implicit APIs. Proxy, node proxy or ambient or CNI are a big enough vocabulary to expose to people.

[sridhargaddam]

The recently merged blog post conveys the following.

...  node’s istio-cni agent, and block pod startup until the agent successfully configures redirection. Since CNI plugins are invoked by the CRI as early as possible in the Kubernetes pod creation process, this ensures that we can establish traffic redirection early enough to prevent traffic escaping during startup, without relying on things like init containers.

Can you please explain the race condition that this PR is addressing?

[bleggett]

The recently merged blog post conveys the following.

...  node’s istio-cni agent, and block pod startup until the agent successfully configures redirection. Since CNI plugins are invoked by the CRI as early as possible in the Kubernetes pod creation process, this ensures that we can establish traffic redirection early enough to prevent traffic escaping during startup, without relying on things like init containers.

Can you please explain the race condition that this PR is addressing?

If the CNI agent is already installed on a node before any pods are scheduled on that node, it works like the blog.

If, because you might be scaling up new nodes in an existing cluster, whatever platform you are using happens to schedule pods on those new nodes without waiting for the Istio CNI agent to be started and the plugin to be installed on the node, then the plugin cannot block startup.

The plugin has to be there to block startup, and there are some existing-cluster node scaling scenarios where pods might end up on a node before the CNI agent and the plugin do - and that's the race this is fixing.

[sridhargaddam]

Can you please explain the race condition that this PR is addressing?

If the CNI agent is already installed on a node before any pods are scheduled on that node, it works like the blog.

If, because you might be scaling up new nodes in an existing cluster, whatever platform you are using happens to schedule pods on those new nodes without waiting for the Istio CNI agent to be started and the plugin to be installed on the node, then the plugin cannot block startup.

The plugin has to be there to block startup, and there are some existing-cluster node scaling scenarios where pods might end up on a node before the CNI agent and the plugin do - and that's the race this is fixing.

Thanks for the detailed explanation @bleggett

[zengyuxing007]

While I understand that this approach solves the problem of the business pod waiting for the ztunnel to be ready, the init container is not a good way to do it. In a way it looks like Ambient is going back to sidecar mode.

[bleggett]

While I understand that this approach solves the problem of the business pod waiting for the ztunnel to be ready, the init container is not a good way to do it. In a way it looks like Ambient is going back to sidecar mode.

While it's not really going back to sidecar mode, as sidecars never were able to fully solve this CNI readiness problem, I tend to agree on the optics, which is why, for people that need to manage this, #48818 is likely preferable.

This is a fallback for users that can't or won't use the untaint controller.

[zengyuxing007]

While I understand that this approach solves the problem of the business pod waiting for the ztunnel to be ready, the init container is not a good way to do it. In a way it looks like Ambient is going back to sidecar mode.

While it's not really going back to sidecar mode, as sidecars never were able to fully solve this CNI readiness problem, I tend to agree on the optics, which is why, for people that need to manage this, #48818 is likely preferable.

This is a fallback for users that can't or won't use the untaint controller.

I agree with u , the untaint controller is preferable~ But if we use the initContainer approach, I think it would be better to leave the iptables execution logic to ztunnel (instead of istio-cni), since consistency is easier to achieve.

[istio-testing]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[linsun]

Hi @yuval-k let me know if you plan to update the PR to latest?

Any objection on this approach @hzxuzhonghu or @keithmattix or @howardjohn?

[linsun]

While I understand that this approach solves the problem of the business pod waiting for the ztunnel to be ready, the init container is not a good way to do it. In a way it looks like Ambient is going back to sidecar mode.

While it's not really going back to sidecar mode, as sidecars never were able to fully solve this CNI readiness problem, I tend to agree on the optics, which is why, for people that need to manage this, #48818 is likely preferable.
This is a fallback for users that can't or won't use the untaint controller.

I agree with u , the untaint controller is preferable~ But if we use the initContainer approach, I think it would be better to leave the iptables execution logic to ztunnel (instead of istio-cni), since consistency is easier to achieve.

It'd be up to the user/provider to decide which approach (untaint or use this initcontainer approach). What we are providing is the 2nd choice as we have heard from the community some folks don't like untaint.

[howardjohn]

Alternative idea: what if we try out the 'device plugin' idea for ambient here? #40303 (comment)

Its superior to this method if it works since it has no runtime cost; init containers are slow and add non-trivial latency to pod startup (can be seconds, which can be 2x for fast containers)

I don't mind this as well if its ready to go and we want to followup no device plugin long term. It doesn't need to block.

Note: device plugin would work with sidecars, too, but it might be a good opportunity to try the slightly riskier approach on new code vs old code

[bleggett]

Alternative idea: what if we try out the 'device plugin' idea for ambient here? #40303 (comment)

Its superior to this method if it works since it has no runtime cost; init containers are slow and add non-trivial latency to pod startup (can be seconds, which can be 2x for fast containers)

I don't mind this as well if its ready to go and we want to followup no device plugin long term. It doesn't need to block.

Note: device plugin would work with sidecars, too, but it might be a good opportunity to try the slightly riskier approach on new code vs old code

I agree it's less invasive than init containers. We could probably reuse the node tainter as the DRA driver, and let you pick either method by configuring the same component.