ambient mesh doesn't support docker desktop kubernetes #47436

chingkt · 2023-10-18T19:33:39Z

Is this the right place to submit this?

This is not a security vulnerability or a crashing bug
This is not a question about how to use Istio

Bug Description

Docker desktop uses kubenet as the CNI plugin by default, which supports veth, which is required to run ambient mesh. So theoretically, it should already support ambient mesh. However, some user complained not being able to use it. I have tried this official tutorial but failed at the application deployment step. Error messages from the istio-cni-node:

2023-10-18T17:48:15.521939Z     error   ambient Failed to list ipset entries: failed to list ipset ztunnel-pods-ips: no such file or directory
2023-10-18T17:48:15.522510Z     info    ambient Pod 'notsleep-5ccf57569-g54sh/default' (9411ba7d-70bf-4981-a958-f92bea4cdcaa) is not in ipset
2023-10-18T17:48:15.553004Z     error   ambient Failed to list ipset entries: failed to list ipset ztunnel-pods-ips: no such file or directory```

### Version

```Text
$istioctl version
client version: 1.19.0
control plane version: 1.19.0
data plane version: 1.19.0 (1 proxies)

$kubectl version --short              
Client Version: v1.27.1
Kustomize Version: v5.0.1
Server Version: v1.27.2

Additional Information

bug-report.tar.gz

The text was updated successfully, but these errors were encountered:

bleggett · 2023-10-18T19:37:31Z

There's an open issue with docker desktop around lack of support for CNI plugins generally (including e.g. Cilium): docker/roadmap#311

I believe this is because of the specific bridge CNI mode Docker Desktop kube uses, e.g:

root@docker-desktop:/host/etc/cni/net.d# cat 10-default.conflist 
{
  "cniVersion": "0.3.1",
  "name": "default",
  "plugins": [
    {
      "bridge": "cni0",
      "dns": {
        "nameservers": [
          "10.1.0.1"
        ]
      },
      "hairpinMode": true,
      "ipMasq": true,
      "ipam": {
        "gateway": "10.1.0.1",
        "subnet": "10.1.0.0/16",
        "type": "host-local"
      },
      "isDefaultGateway": true,
      "type": "bridge"
    },
    {
      "capabilities": {
        "portMappings": true
      },
      "snat": true,
      "type": "portmap"
    }
  ]
}

bleggett · 2023-10-18T22:54:18Z

The other error mentioned/seen:

│ Warning FailedCreatePodSandBox 14s kubelet Failed to create pod sandbox: rpc error: code = Un ││ known desc = [failed to set up sandbox container "b7493d90e9d5e93f486a74cef208c72fc7b8ba9fb54845c0caa28cb4e460b2db" networ │ k for pod "httpbin-86869bccff-pq7vj": networkPlugin cni failed to set up pod "httpbin-86869bccff-pq7vj_default" network: p ││ lugin type="loopback" failed (add): missing network name:, failed to clean up sandbox container "b7493d90e9d5e93f486a74cef ││ 208c72fc7b8ba9fb54845c0caa28cb4e460b2db" network for pod "httpbin-86869bccff-pq7vj": networkPlugin cni failed to teardown ││ pod "httpbin-86869bccff-pq7vj_default" network: plugin type="loopback" failed (delete): missing network name] ││ Normal SandboxChanged 0s (x2 over 14s) kubelet

This seems to be because docker-desktop uses a loopback plugin at lowest priority that runs last, and doesn't actually support the current CNI version spec?

linsun · 2023-12-20T15:53:52Z

Hi @chingkt - thank you for reporting this! it is a good time to evaluate this again, I believe with the latest istio-cni change #48212, ambient for docker desktop k8s should be supported. While the PRs are still under review, can you try it following the instructions below?

#48212 (comment)

linsun · 2024-01-29T00:13:35Z

This is resolved, please try latest master or release 1.21 build, or wait for the official 1.21 release. see #48212

linsun · 2024-01-29T15:08:12Z

Sorry, just got confirmation from Eric that the change is not in beta 0 yet, but should be in -beta.1 since it was merged after beta.0 was released. I expect that the new build will be available in a day or 2.

chingkt · 2024-02-02T18:39:06Z

@linsun A big thank you to the Istio team for your effort!

harsh4870 · 2024-02-04T20:30:03Z

Thank you team for help & support however seeing issue with Ambient Beta-1 release Docker Desktop M1

kubectl -n istio-system get pods                  
NAME                                    READY   STATUS              RESTARTS   AGE
istio-cni-node-8njst                    1/1     Running             0          116s
istio-ingressgateway-689f9d6fb4-4qssc   1/1     Running             0          20m
istiod-556d7d4cf5-ndg7m                 1/1     Running             0          20m
ztunnel-78ldc                           0/1     Terminating         0          20m
ztunnel-zx8m7                           0/1     ContainerCreating   0          54s

Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               6m28s                 default-scheduler  Successfully assigned istio-system/ztunnel-zx8m7 to docker-desktop
  Warning  FailedCreatePodSandBox  6m27s                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "c88ecf6787ad644d31272bff249c3a6320a86cb5734dc975fd08169f25fb6442" network for pod "ztunnel-zx8m7": networkPlugin cni failed to set up pod "ztunnel-zx8m7_istio-system" network: plugin type="loopback" failed (add): missing network name:, failed to clean up sandbox container "c88ecf6787ad644d31272bff249c3a6320a86cb5734dc975fd08169f25fb6442" network for pod "ztunnel-zx8m7": networkPlugin cni failed to teardown pod "ztunnel-zx8m7_istio-system" network: plugin type="loopback" failed (delete): missing network name]

harsh4870 · 2024-02-05T15:51:37Z

Created issue - #49208

gjycn · 2024-03-09T10:06:42Z

Thank you team for help & support however seeing issue with Ambient Beta-1 release Docker Desktop M1

kubectl -n istio-system get pods                  
NAME                                    READY   STATUS              RESTARTS   AGE
istio-cni-node-8njst                    1/1     Running             0          116s
istio-ingressgateway-689f9d6fb4-4qssc   1/1     Running             0          20m
istiod-556d7d4cf5-ndg7m                 1/1     Running             0          20m
ztunnel-78ldc                           0/1     Terminating         0          20m
ztunnel-zx8m7                           0/1     ContainerCreating   0          54s

Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               6m28s                 default-scheduler  Successfully assigned istio-system/ztunnel-zx8m7 to docker-desktop
  Warning  FailedCreatePodSandBox  6m27s                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "c88ecf6787ad644d31272bff249c3a6320a86cb5734dc975fd08169f25fb6442" network for pod "ztunnel-zx8m7": networkPlugin cni failed to set up pod "ztunnel-zx8m7_istio-system" network: plugin type="loopback" failed (add): missing network name:, failed to clean up sandbox container "c88ecf6787ad644d31272bff249c3a6320a86cb5734dc975fd08169f25fb6442" network for pod "ztunnel-zx8m7": networkPlugin cni failed to teardown pod "ztunnel-zx8m7_istio-system" network: plugin type="loopback" failed (delete): missing network name]

same issue on Windows 11, docker-desktop 4.28.0 base wsl2, linux kernel is "6.1.21.2-microsoft-standard-WSL2+", k8s version is "1.29.1", istio version is "1.21.0-rc.1"

bleggett · 2024-03-11T16:37:27Z

Thank you team for help & support however seeing issue with Ambient Beta-1 release Docker Desktop M1

kubectl -n istio-system get pods                  
NAME                                    READY   STATUS              RESTARTS   AGE
istio-cni-node-8njst                    1/1     Running             0          116s
istio-ingressgateway-689f9d6fb4-4qssc   1/1     Running             0          20m
istiod-556d7d4cf5-ndg7m                 1/1     Running             0          20m
ztunnel-78ldc                           0/1     Terminating         0          20m
ztunnel-zx8m7                           0/1     ContainerCreating   0          54s

Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               6m28s                 default-scheduler  Successfully assigned istio-system/ztunnel-zx8m7 to docker-desktop
  Warning  FailedCreatePodSandBox  6m27s                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "c88ecf6787ad644d31272bff249c3a6320a86cb5734dc975fd08169f25fb6442" network for pod "ztunnel-zx8m7": networkPlugin cni failed to set up pod "ztunnel-zx8m7_istio-system" network: plugin type="loopback" failed (add): missing network name:, failed to clean up sandbox container "c88ecf6787ad644d31272bff249c3a6320a86cb5734dc975fd08169f25fb6442" network for pod "ztunnel-zx8m7": networkPlugin cni failed to teardown pod "ztunnel-zx8m7_istio-system" network: plugin type="loopback" failed (delete): missing network name]

same issue on Windows 11, docker-desktop 4.28.0 base wsl2, linux kernel is "6.1.21.2-microsoft-standard-WSL2+", k8s version is "1.29.1", istio version is "1.21.0-rc.1"

Yes, as I mentioned above, this is a general issue with docker desktop and CNI plugins: #47436 (comment)

docker desktop uses broken CNI plugins that do not correctly support the CNI spec, and as such do not properly support custom CNIs, see also docker/for-mac#4626

docker desktop's support for CNI plugins is broken.
Use minikube or kind or anything else, and you won't have this problem. docker desktop is not sufficient for anything other than a subset of basic/trivial networking setups. If you can repro this in any other environment, please reopen.
Raise an issue with docker desktop around their general support for CNI plugins.

istio-policy-bot added area/ambient Issues related to ambient mesh area/environments area/test and release area/user experience labels Oct 18, 2023

chingkt mentioned this issue Oct 18, 2023

Allow custom CNI plugins in Docker Desktop Kubernetes docker/roadmap#311

Open

linsun closed this as completed Jan 29, 2024

bleggett mentioned this issue Feb 5, 2024

istio-cni failing with Docker Desktop on Macbook M1 #49208

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ambient mesh doesn't support docker desktop kubernetes #47436

ambient mesh doesn't support docker desktop kubernetes #47436

chingkt commented Oct 18, 2023 •

edited by istio-policy-bot

Loading

bleggett commented Oct 18, 2023 •

edited

Loading

bleggett commented Oct 18, 2023 •

edited

Loading

linsun commented Dec 20, 2023

linsun commented Jan 29, 2024

linsun commented Jan 29, 2024

chingkt commented Feb 2, 2024

harsh4870 commented Feb 4, 2024 •

edited

Loading

harsh4870 commented Feb 5, 2024

gjycn commented Mar 9, 2024

bleggett commented Mar 11, 2024 •

edited

Loading

ambient mesh doesn't support docker desktop kubernetes #47436

ambient mesh doesn't support docker desktop kubernetes #47436

Comments

chingkt commented Oct 18, 2023 • edited by istio-policy-bot Loading

Is this the right place to submit this?

Bug Description

Additional Information

bleggett commented Oct 18, 2023 • edited Loading

bleggett commented Oct 18, 2023 • edited Loading

linsun commented Dec 20, 2023

linsun commented Jan 29, 2024

linsun commented Jan 29, 2024

chingkt commented Feb 2, 2024

harsh4870 commented Feb 4, 2024 • edited Loading

harsh4870 commented Feb 5, 2024

gjycn commented Mar 9, 2024

bleggett commented Mar 11, 2024 • edited Loading

chingkt commented Oct 18, 2023 •

edited by istio-policy-bot

Loading

bleggett commented Oct 18, 2023 •

edited

Loading

bleggett commented Oct 18, 2023 •

edited

Loading

harsh4870 commented Feb 4, 2024 •

edited

Loading

bleggett commented Mar 11, 2024 •

edited

Loading