Upstream solo's implementation of inpod redirection between ztunnel and app pod

[linsun]

In today's ambient WG meeting, @yuval-k and @bleggett presented an innovative solution to redirect traffic from ztunnel to workload pod within the pod's network namespace. In-pod Traffic tradirection model proposal (ambient that works on any CNI)
https://docs.google.com/document/d/1dynKlnNgIOywv3cwuw_2RCk_SKFRs7IrESaC8_r-sm0

The next step is:

• open ztunnel and istio PRs for the work
• provide a repo where folks can try it before the code is merged.

@yuval-k will open the PR on ztunnel side, @bleggett will open the PR on istio-cni side.

[linsun]

not sure why i can't assign @yuval-k...

[ericvn]

not sure why i can't assign @yuval-k...

It looks like the invite was sent to join the org, but it has not yet been accepted. Have them look at their main GitHub page and see if there is an invite.

[linsun]

That was it, thank you @ericvn !

[hzxuzhonghu]

After reading through it, looks like a very ingenious and innovative design. Ztunnel traffic handling will be within each pod's netns just like sidecar.

What confuse me is for inbound traffic, is the traffic intercepeted to ztunnel socket within pod netns? Assume it is not, then a direct pod/svc access from non-ambient pod will target to the real port of application, so authn/authz will be both skipped

[linsun]

Thanks Zhonghu for the feedback. Yes, for inbound traffic, it will be intercepted within pod network ns thus the authz/n policies can continue to be enforced. Basically, this works exactly same as sidecars with HBONE enabled where you'd have to adjust port to accomodate HBONE port 15008 if your authz has port in it.

cc @bleggett and @yuval-k who knows more than me to chime in.

[linsun]

#48253 is the CNI PR from @bleggett

[bleggett]

What confuse me is for inbound traffic, is the traffic intercepeted to ztunnel socket within pod netns? Assume it is not, then a direct pod/svc access from non-ambient pod will target to the real port of application, so authn/authz will be both skipped

Inbound is intercepted within pod netns, @linsun is correct.

[linsun]

Copy instruction from @yuval-k from slack to GitHub for easier reference:

To run a demo for inpod ztunnel redirection we discussed on Wed:

istioctl install -d manifests/ --set hub=gcr.io/solo-test-236622/inpod-istio --set tag=1.19.4-1 -y --set profile=ambient -f istio-cni.yml --set values.global.logging.level="all:debug"

kubectl label namespace default istio.io/dataplane-mode=ambient

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.19/samples/bookinfo/platform/kube/bookinfo.yaml

see manifests and istio-cni.yaml in attached zip. Note that you may have issues using ARM64 macs; so use intel machines.
You can see the ztunnel sockets in the pod like so:

kubectl debug $(kubectl get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}') -it --image jrecord/nettools -- ss -ntlp

inpod.zip

[yuval-k]

p.o.c branch for egress bypass mitigation https://github.com/yuval-k/istio/tree/ebpf-bind-sidebranch

[linsun]

A couple of folks pinged me for update on this so I'm updating github:

The ztunnel PR is ready to go hopefully @howardjohn will approve it today. The istio cni PR is still under 2nd round of reviews with all comments are addressed. With code freeze coming in Istio 1.21 in a few days, we are confident the ztunnel PR will be merged in, and hopefully the istio cni PR shortly after so both will land in 1.21.

cc @istio/release-managers FYI

[zengyuxing007]

This solution is a great idea, but there may be some issues with landing it, especially in production environments. I would like to know how to upgrade ztunnel seamlessly （#48387）, there is always a switching procedure for in-place upgrades, during which business requests may fail.

[linsun]

@zengyuxing007 I read your comment and understand your concern on upgrading ztunnel without down time. Note, that is a separate issue from the in-pod contribution from Solo to Istio - feel free to open a feature request and design proposal for it.

[zengyuxing007]

@linsun ok, thanks!

open the issue : #48949

[linsun]

All approvers are in - we expect this to land in 1.21!

[linsun]

once #49028 is merged, we should be able to close this

[hzxuzhonghu]

done

[linsun]

Just got confirmation from Eric that the change is not in beta 0 yet, but should be in -beta.1 since it was merged after beta.0 was released. I expect that the new build will be available in a day or 2.