cannot create new pod in an ambient labeled namespace in ebpf mode

[KfreeZ]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

I have an ambient mesh setup in ebpf redirection mode,
It works when I following this sequence,

1. install workload (sleep, bookinfo)
2. label the namespace(default) as ambient
3. send the traffic

But when I restart the application pod in the namespace (default), I encountered below error:

│   Warning  FailedCreatePodSandBox  15m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c3d3ea56d361023f34 │
│ 22fc4fb5258c48d0eb8594a25df3c1c8d35a318e8a33b3" network for pod "sleep-7656cf8794-qc7f8": networkPlugin cni failed to set up pod "sleep-7656cf8794-qc7f8_default" network: plugin type="istio-cn │
│ i" name="istio-cni" failed (add): failed to get info for if(eth0) in ns(net): failed to Statfs "/var/run/netns/net": no such file or directory

Version

[root@k8s-node02 ~]# docker version
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:09:18 2023
OS/Arch: linux/amd64
Context: default

Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:08:20 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.24
GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc
docker-init:
Version: 0.19.0
GitCommit: de40ad0

[root@k8s-master bin]# ./istioctl version
client version: 1.20.0
control plane version: 1.20.0
data plane version: 1.20.0 (4 proxies)

Additional Information

No response

[sword-jin]

Interesting, could you provide some context and could this bug be reproduced?

which pod was restarted? and where is the error message from?

[KfreeZ]

Interesting, could you provide some context and could this bug be reproduced?

which pod was restarted? and where is the error message from?

It's 100% reproducible, my CRI sandbox is cri-dockerd, CNI is calico 3.26.3, any pod cannot be successfully created if the namespace is injected by ambient, the error log is the retrieved from the pod's log, it can also be found in kubelet's journal.

[sword-jin]

Interesting, could you provide some context and could this bug be reproduced?
which pod was restarted? and where is the error message from?

It's 100% reproducible, my CRI sandbox is cri-dockerd, CNI is calico 3.26.3, any pod cannot be successfully created if the namespace is injected by ambient, the error log is the retrieved from the pod's log, it can also be found in kubelet's journal.

Try to stop calico, then restart, I think the problem may be from calico, something incompatible. I will give it a try

[hzxuzhonghu]

Is ebpf mode comptabible with calico now, there is a issue saying calico can not work with ambient yet

[hanxiaop]

Is ebpf mode comptabible with calico now, there is a issue saying calico can not work with ambient yet

Calico works well with Istio Ambient in eBPF mode; however, in normal mode, it has network issues.

[hanxiaop]

@KfreeZ Have you followed these steps? https://github.com/istio/istio/tree/master/manifests/charts/istio-cni#calico

[KfreeZ]

@hzxuzhonghu @hanxiaop yes, eBPF mode is "half working" for me with the latest istio(1.20) and the kubectl patch felixconfigurations default --type='json' -p='[{"op": "add", "path": "/spec/workloadSourceSpoofing", "value": "Any"}] for now.
I have confirmed the traffic is re-directed to ztunnel and ztunnel is proxying to the right endpoint.
My problem is I cannot restart the app pod or create new pod because once the namespace is injected with ambient, the CRI sandbox would have some issue when it's calling the CNI, the error log is attached above.
More specifically, I believe the problem exists between the sandbox(cri-dockerd) and istio cni because the "/var/run/netns/net" in error log is not reasonable for me.

[PlatformLC]

I'm working on this and have found some clues. Will update later.