Docker Swarm Overlay encryption is broken in WSL2

[ACoderLife]

Windows Version

Microsoft Windows [Version 10.0.22621.1555]

WSL Version

1.2.5.0

Are you using WSL 1 or WSL 2?

• WSL 2
• WSL 1

Kernel Version

5.15.90.1

Distro Version

No response

Other Software

Docker Desktop (Windows) 4.18.0 (104112)

Repro Steps

using a compose file with an encrypted overlay network like this:
version: '3.8'

services:
nginx:
image: nginx:latest
ports:
- "8081:80"

networks:
default:
driver: overlay
attachable: true
driver_opts:
encrypted: ''

in docker swarm start it as a service:
docker stack deploy -c docker-compose.yml test

Expected Behavior

This use to work in an older version of Docker Desktop.

Actual Behavior

The service is not starting and when I inspect using:
docker stack ps --no-trunc test

I am getting the following errors:
ID NAME IMAGE
NODE DESIRED STATE CURRENT STATE ERROR PORTS
e678y0f1lns28mft1bbz2ok66 test_nginx.1 nginx:latest@sha256:63b44e8ddb83d5dd8020327c1f40436e37a6fffd3ef2498a6204df23be6e7e94 docker-desktop Ready Rejected less than a second ago "network sandbox join failed: subnet sandbox join failed for "10.0.22.0/24": 2 errors occurred:
* could not install mangle rule: (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1054208 -j MARK --set-mark 13681891: iptables v1.8.7 (legacy): Couldn't load match `u32':No such file or directory

Try iptables -h' or 'iptables --help' for more information. (exit status 2)) * could not install mangle rule: (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m bpf --bytecode 6,177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1054208,6 0 0 0,6 0 0 4294967295 -j MARK --set-mark 13681891: iptables v1.8.7 (legacy): Couldn't load match bpf':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
(exit status 2))

"
wylyqfg2n6z5w69e49yz35ikj _ test_nginx.1 nginx:latest@sha256:63b44e8ddb83d5dd8020327c1f40436e37a6fffd3ef2498a6204df23be6e7e94 docker-desktop Shutdown Rejected 5 seconds ago "network sandbox join failed: subnet sandbox join failed for "10.0.22.0/24": 2 errors occurred:
* could not install mangle rule: (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1054208 -j MARK --set-mark 13681891: iptables v1.8.7 (legacy): Couldn't load match `u32':No such file or directory

Try iptables -h' or 'iptables --help' for more information. (exit status 2)) * could not install mangle rule: (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m bpf --bytecode 6,177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1054208,6 0 0 0,6 0 0 4294967295 -j MARK --set-mark 13681891: iptables v1.8.7 (legacy): Couldn't load match bpf':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
(exit status 2))

"
80gpjclyy4uryyilwyyhdspe6 _ test_nginx.1 nginx:latest@sha256:63b44e8ddb83d5dd8020327c1f40436e37a6fffd3ef2498a6204df23be6e7e94 docker-desktop Shutdown Rejected 5 seconds ago "network sandbox join failed: subnet sandbox join failed for "10.0.22.0/24": 2 errors occurred:
* could not install mangle rule: (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1054208 -j MARK --set-mark 13681891: iptables v1.8.7 (legacy): Couldn't load match `u32':No such file or directory

Try iptables -h' or 'iptables --help' for more information. (exit status 2)) * could not install mangle rule: (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m bpf --bytecode 6,177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1054208,6 0 0 0,6 0 0 4294967295 -j MARK --set-mark 13681891: iptables v1.8.7 (legacy): Couldn't load match bpf':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
(exit status 2))

Diagnostic Logs

No response

[LauJosefsen]

This is definitely a problem, and seemingly not docker desktop, as it also happens with docker in Ubuntu in WSL2.

We now have 2 separate machines at my work experiencing this problem.

We can replicate by:

1. Clean installing windows from latest ISO.
2. Install WSL2 latest version.
3. Install Ubuntu 22.04
4. Install docker using https://docs.docker.com/engine/install/ubuntu/
5. Try to deploy an encrypted overlay network, get above mentioned error.

[ACoderLife]

I think it is related to:
#9946
With comment: "WSL kernels are build with CONFIG_NETFILTER_XT_MATCH_U32 is not set."

But this was working in older builds.

[LauJosefsen]

This started with docker 23.0.3 i believe. Downgrading to docker 23.0.2 fixes for now.

[neersighted]

Hi all, Moby (Docker) maintainer here.

This is not a bug in Moby. Or rather, users are now experiencing this because we fixed a bug in Moby. Versions >= 23.0.3, >= 20.10.24 include a security patch in order to correctly fail to created encrypted overlay networks when encountering a lack of kernel support. Previous versions of Moby were silently failing to encrypt data; you thought it 'worked' because it passed data, but that data was in fact unencrypted.

Please review these CVEs for more details:

• CVE-2023-28840: Encrypted overlay network may be unauthenticated
• CVE-2023-28841: Encrypted overlay network traffic may be unencrypted
• CVE-2023-28842: Encrypted overlay network with a single endpoint is unauthenticated

The WSL2 kernel will need to be built with the xt_u32 or xt_bpf kernel modules (they don't need to be loadable; built-in is fine) in order to support Swarm overlays with encryption. xt_bpf would be strongly preferred as we plan to remove usage of xt_u32 in the near future.

[ACoderLife]

Will anyone on the WSL team give an indication of xt_bpf being added to the kernel by default? Are WSL interested in providing a network encryption option that is available by default? Has Docker got an internal ticket from MS on this?