Docker-Desktop 4.27.0 breaks Istio on Kubernetes #7148

xvzf · 2024-01-29T09:16:53Z

Description

After the upgrade to 4.27.0 Istio running on-top of Docker Desktop Kubernetes stops working on the istio-init fails to start.

Here are the logs of one of the failing init containers:

2024-01-29T09:09:56.603820Z	info	Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=true
INVALID_DROP=

2024-01-29T09:09:56.603868Z	info	Istio iptables variables:
IPTABLES_VERSION=
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=5671,5672,25672
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DUAL_STACK=false
DNS_CAPTURE=true
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[10.96.0.10],[]
NETWORK_NAMESPACE=
CNI_MODE=false
EXCLUDE_INTERFACES=

2024-01-29T09:09:56.603950Z	info	Running iptables-restore with the following input:
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -p tcp --dport 5671 -j RETURN
-A ISTIO_OUTPUT -p tcp --dport 5672 -j RETURN
-A ISTIO_OUTPUT -p tcp --dport 25672 -j RETURN
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp -m multiport ! --dports 53,15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp ! --dport 53 -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp ! --dport 53 -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -p tcp --dport 53 -d 10.96.0.10/32 -j REDIRECT --to-ports 15053
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A OUTPUT -p udp --dport 53 -m owner --uid-owner 1337 -j RETURN
-A OUTPUT -p udp --dport 53 -m owner --gid-owner 1337 -j RETURN
-A OUTPUT -p udp --dport 53 -d 10.96.0.10/32 -j REDIRECT --to-port 15053
COMMIT
* raw
-A OUTPUT -p udp --dport 53 -m owner --uid-owner 1337 -j CT --zone 1
-A OUTPUT -p udp --sport 15053 -m owner --uid-owner 1337 -j CT --zone 2
-A OUTPUT -p udp --dport 53 -m owner --gid-owner 1337 -j CT --zone 1
-A OUTPUT -p udp --sport 15053 -m owner --gid-owner 1337 -j CT --zone 2
-A OUTPUT -p udp --dport 53 -d 10.96.0.10/32 -j CT --zone 2
-A PREROUTING -p udp --sport 53 -s 10.96.0.10/32 -j CT --zone 1
COMMIT
2024-01-29T09:09:56.604012Z	info	Running command (with wait lock): iptables-restore --noflush --wait=30
2024-01-29T09:09:56.604934Z	error	Command error output: xtables other problem: line 2 failed
2024-01-29T09:09:56.605120Z	info	Running command (without lock): iptables-save 
2024-01-29T09:09:56.606686Z	info	Command output: 
# Generated by iptables-save v1.8.7 on Mon Jan 29 09:09:56 2024
*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Mon Jan 29 09:09:56 2024
# Generated by iptables-save v1.8.7 on Mon Jan 29 09:09:56 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A OUTPUT -p udp -m udp --dport 53 -m owner --uid-owner 1337 -j RETURN
-A OUTPUT -p udp -m udp --dport 53 -m owner --gid-owner 1337 -j RETURN
-A OUTPUT -d 10.96.0.10/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m tcp --dport 5671 -j RETURN
-A ISTIO_OUTPUT -p tcp -m tcp --dport 5672 -j RETURN
-A ISTIO_OUTPUT -p tcp -m tcp --dport 25672 -j RETURN
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m multiport ! --dports 53,15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 10.96.0.10/32 -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Mon Jan 29 09:09:56 2024

2024-01-29T09:09:56.606706Z	error	exit status 1

Reproduce

istioctl install
Start any pod with the istio sidecar enabled

Expected behavior

Istio works fine like in all previous releases

docker version

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:06:12 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.27.0 (135262)
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:35 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    25.0.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /Users/xvzf/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.3-desktop.1
    Path:     /Users/xvzf/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.22
    Path:     /Users/xvzf/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/xvzf/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/xvzf/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/xvzf/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /Users/xvzf/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/xvzf/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/xvzf/.docker/cli-plugins/docker-scout

Server:
 Containers: 127
  Running: 68
  Paused: 0
  Stopped: 59
 Images: 47
 Server Version: 25.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.756GiB
 Name: docker-desktop
 ID: b3867f20-46f6-4d96-bb14-2840a648273a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

89814F6E-F5C8-4F6E-BCA8-FE138807A3C4/20240129091521

Additional Info

No response

The text was updated successfully, but these errors were encountered:

xvzf · 2024-01-29T09:17:17Z

Note: this is also affecting Docker Desktop for Linux.

xvzf · 2024-01-29T09:44:05Z

The culprit is the kernel configuration, which disables a bunch of conntrack features required to run istio and other service meshes:

-CONFIG_NF_CONNTRACK_ZONES=y
-CONFIG_NF_CONNTRACK_PROCFS=y
-CONFIG_NF_CONNTRACK_EVENTS=y
-CONFIG_NF_CONNTRACK_TIMEOUT=y
-CONFIG_NF_CONNTRACK_TIMESTAMP=y
+# CONFIG_NF_CONNTRACK_ZONES is not set
+# CONFIG_NF_CONNTRACK_PROCFS is not set
+# CONFIG_NF_CONNTRACK_EVENTS is not set
+# CONFIG_NF_CONNTRACK_TIMEOUT is not set
+# CONFIG_NF_CONNTRACK_TIMESTAMP is not set

For the record, the full diff can be found here: https://gist.github.com/xvzf/7a81f881033cab9a930f081929c3c5f0

dgageot · 2024-01-29T18:30:27Z

Thanks a lot @xvzf for fine the root cause. I'll add them back and add a few tests specific to istio.

dgageot · 2024-01-29T18:41:38Z

@xvzf do you know if only those 5 are required? How can I quickly test Istio nowadays? I haven't tried in a long time.

xvzf · 2024-01-30T08:44:40Z

Hi @dgageot, thanks for looking into this!

There's a platform prerequisites section in the Istio documentation but no specific configuration options for the modules (e.g. zone support on conntrack).

As for testing this specific bug, it should be sufficient to:

Get a copy of istioctl ()
Install istio e.g. with the demo profile using istioctl install --set profile=demo
Label e.g. the default namespace with istio-injection=enabled
Start any pod in the default namespace
Validate the istio-init (init) container and istio-proxy container both come up

Also a suggestion form my colleague @jordiclariana:

docker run --rm -ti --cap-add NET_ADMIN gcr.io/istio-release/proxyv2:1.20.2 istio-iptables --redirect-dns

works on 4.61.1 but fails on 4.27.0. Might be worth adding it to your test-suite

dgageot · 2024-02-01T18:05:43Z

Hi @xvzf! Docker Desktop 4.27.1 is out. Could you tell me if it solves your issue?

dgageot · 2024-02-01T18:46:19Z

@xvzf Sorry for the false information. I just learnt that those patches will ship with 4.27.2

craigbox · 2024-02-12T23:27:41Z

David is back at Docker - hooray!

dgageot · 2024-02-13T09:36:42Z

David is back at Docker - hooray!

Indeed :-) Have you tried to start Istio with Docker Desktop 4.27.2, Craig?

craigbox · 2024-02-13T09:56:21Z

No, I just chanced into this link in the Istio slack. I happen to have upgraded to 4.27.1 today but I had to do it manually - as the release notes suggested.

jordiclariana · 2024-02-13T10:09:41Z

@dgageot , we did try 4.27.2 with Istio and so far so good. Seems that the problem is solved there. Thanks!

dgageot · 2024-02-13T10:36:57Z

\o/ Thanks @jordiclariana. I'm closing the issue then

xvzf added the needs-triage label Jan 29, 2024

docker-robot bot added the version/4.27.0 label Jan 29, 2024

dgageot self-assigned this Jan 29, 2024

dgageot added area/kernel Linux kernel bug kind/regression status/triage and removed needs-triage labels Jan 29, 2024

This was referenced Jan 29, 2024

openvswitch changed to built-in kernel module in Docker Desktop 4.27.0 #7151

Closed

Antrea no longer works with Kind after upgrading to Docker Desktop 4.27.0 antrea-io/antrea#5939

Closed

dgageot closed this as completed Feb 13, 2024

dgageot removed their assignment Feb 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker-Desktop 4.27.0 breaks Istio on Kubernetes #7148

Docker-Desktop 4.27.0 breaks Istio on Kubernetes #7148

xvzf commented Jan 29, 2024

xvzf commented Jan 29, 2024

xvzf commented Jan 29, 2024

dgageot commented Jan 29, 2024

dgageot commented Jan 29, 2024

xvzf commented Jan 30, 2024

dgageot commented Feb 1, 2024

dgageot commented Feb 1, 2024

craigbox commented Feb 12, 2024

dgageot commented Feb 13, 2024

craigbox commented Feb 13, 2024 via email

jordiclariana commented Feb 13, 2024

dgageot commented Feb 13, 2024

Docker-Desktop 4.27.0 breaks Istio on Kubernetes #7148

Docker-Desktop 4.27.0 breaks Istio on Kubernetes #7148

Comments

xvzf commented Jan 29, 2024

Description

Reproduce

Expected behavior

docker version

docker info

Diagnostics ID

Additional Info

xvzf commented Jan 29, 2024

xvzf commented Jan 29, 2024

dgageot commented Jan 29, 2024

dgageot commented Jan 29, 2024

xvzf commented Jan 30, 2024

dgageot commented Feb 1, 2024

dgageot commented Feb 1, 2024

craigbox commented Feb 12, 2024

dgageot commented Feb 13, 2024

craigbox commented Feb 13, 2024 via email

jordiclariana commented Feb 13, 2024

dgageot commented Feb 13, 2024