Docker swarm requires more iptables extensions

[matfax]

Describe the bug

Not all features of docker swarm work without errors. There is no warning attached, so I presume docker swarm is an expected feature. When dockerd calls legacy iptables with its expected ipvs and state extensions, the rule setup fails. These rules are necessary for overlay networks.

OpenWrt version

r20028-43d71ad93e

OpenWrt target/subtarget

x86/64

Device

Microsoft Corporation Virtual Machine

Image kind

Official downloaded image

Steps to reproduce

opkg update && opkg install docker dockerd kmod-nf-ipvs kmod-nf-vxlan kmod-crypto-gcm kmod-crypto-seqiv kmod-crypto-ghash kmod-ipsec
modprobe ip_vs # or enable as boot module, since it is not loaded by default
/etc/init.d/dockerd enable && /etc/init.d/dockerd start
docker swarm init
docker network create -d overlay --opt encrypted --attachable overnet
docker stack deploy -c hello.yml hello

Content of hello.yml

version: '3.8'

services:
  hello:
    image: 'nginxdemos/hello'
    networks:
      - overnet
    ports:
      - 80:80
      - 443:443
    deploy:
      mode: global

networks:
  overnet:
    external: true

Actual behaviour

Logread shows iptables errors from dockerd.

Failed to add ingress: set up rule failed, [-I DOCKER-INGRESS -m state -p tcp --sport 80 --state ESTABLISHED,RELATED -j ACCEPT]:  (iptables failed: iptables --wait -I DOCKER-INGRESS -m state -p tcp --sport 80 --state ESTABLISHED,RELATED -j ACCEPT: iptables v1.8.7 (legacy): Couldn't load match `state':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))

set up rule failed, [-t nat -A POSTROUTING -m ipvs --ipvs -d 10.0.2.0/24 -j SNAT --to-source 10.0.2.8]: (iptables failed: iptables --wait -t nat -A POSTROUTING -m ipvs --ipvs -d 10.0.2.0/24 -j SNAT --to-source 10.0.2.8: iptables v1.8.7 (legacy): Couldn't load match 'ipvs':No such file or directory\n\nTry 'iptables -h' or 'iptables --help' for more information.\n (exit status 2))"

Failed to add firewall mark rule in sbox lb_i814 (lb-over): reexec failed: exit status 8

Expected behaviour

No error messages are expected from the dockerd package or its swarm feature and its network types.

Additional info

libxt_ipvs.so and libxt_state.so are not present in /usr/lib/ or /usr/lib/iptables/.

The error messages can be difficult to reproduce once the rules are setup or once the networks need to be recreated. It might be necessary to forcefully leave the swarm (i.e. destroy it), and to manually remove the default docker gw bridge network, followed by a restart of the dockerd init.d script or reboot. But the error messages speak for themselves, can be reproduced by iptables -m ipvs and iptables -m state.

docker swarm leave --force
# if it doesn't succeed, restart dockerd and retry leaving
docker network rm docker_gwbridge
reboot

Though I'm using Turris OS, they only clone OpenWRT and apply their own patches. The patches don't affect this issue, as documented by another forum user.

It's possible that other errors occur during a docker swarm setup or with macvlan and ipvlan networks. This issue only concerns the missing extensions for now. The extensions are part of the iptables source and just need to be included in the build.

Diffconfig

No response

Terms

• I am reporting an issue for OpenWrt, not an unsupported fork.

[brada4]

OpenWRT packaging is in good order https://openwrt.org/packages/pkgdata/dockerd
Head back to unsupported fork in regard to make patches to not affect your issue.

[ynezz]

libxt_ipvs.so and libxt_state.so are not present in /usr/lib/ or /usr/lib/iptables/

libxt_ipvs.so would need to be packaged

-m state is deprecated and the legacy aliases are disabled in OpenWrt. You need to replace -m state with -m conntrack and --state with --ctstate

so probably some Docker compat/fix/patch is needed

[brada4]

OpenWRT 21.02.5 r16688

docker - 20.10.17-1
dockerd - 20.10.17-1

Does not complain about iptables at all, it asks for vxlan modules for -doverlay and ipsec modules for --optencrypted.
Obviously to keep it compact many optional/exotic options are not backed with dependencies
After installing random modules rhyming with those i can connect nginx container.

It is blazing difference between TurrisOS and OpenWRT, so you need to convince them to do right packaging for your task.

[matfax]

Does not complain about iptables at all, it asks for vxlan modules for -doverlay and ipsec modules for --optencrypted.

It is not a build bug. Docker doesn't complain because it doesn't expect legacy iptables and OpenWRT's modular package architecture. Moreover, docker normally complains about missing kernel mods, not about missing binary libraries. It would nice if someone patched such checks into docker.

Obviously to keep it compact many optional/exotic options are not backed with dependencies

A very basic swarm setup shouldn't be considered exotic. Swarm is an essential feature of docker and relies on overlay networks for connectivity. I think it's safe to say that a docker user will expect swarm to work when no further warning is given.

After installing random modules rhyming with those i can connect nginx container.

Have you checked the overlay connectivity, not just the swarm joining? The swarm joining isn't affected and docker will exit with a successful join message. But without the conntrack, connectivity issues will occur. And without the ipvs, overlay networks will not be matched by OpenWRT's docker. That unless, I presume, a different method is used for creating and matching rules, such as nftables.

It is blazing difference between TurrisOS and OpenWRT, so you need to convince them to do right packaging for your task.

It's no effort to migrate the patch to TurrisOS. But it's not a TurrisOS-specific issue. It's a simple question of whether these two iptables extensions are to be included in OpenWRT. I'm sure they will serve other purposes as well, outside the docker scope. That's just a feature. But in case that a basic docker swarm setup doesn't work, without vlans or other optional features, it seems to me more like a bug.

[brada4]

The OpenWRT 21 docker package uses conntrack. TurrisOS package is so different that particular issue does not map to OpenWRT.
There is no such thing as optional dependencies in opkg like you have in most desktop linux. Think of average case of 8-16-32MB ROM where having that vxlan for swarm would fill the / denying normal default sandbox mode of docker.

[matfax]

The OpenWRT 21 docker package uses conntrack. TurrisOS package is so different that particular issue does not map to OpenWRT.

Where did you get information that the package is different? I gave your claim the benefit of the doubt, so I checked the Turris repositories and communication. I searched all project files for the term docker. It's only mentioned as part of their CI, so completely unrelated. OpenWRT's repository is cloned during the build process and the applied patches do not modify the docker package.

This is consistent with the communication [1], [2], [3]. According to the communication, there was no effort to integrate docker on its own, they just waited for OpenWRT to include it. The only tweaks necessary/considered were kernel related (e.g. the HudgeTLB control group access). Obviously, Turris OS modifies the build config. But you can't enable something in the config that doesn't exist.

Another argument that supports my assumption that this is OpenWRT-related is that the issue has already been reported by another forum user in 2019, who by the way seemed to use his own build of/or vanilla OpenWRT.

Turris might change other parts of the system and docker might need to adapt to that. The primary question is if the OpenWRT team considers legacy iptables a supported feature or 'exotic and optional'. If there's no support for iptables anymore, I just wonder why the package still exists.

There is no such thing as optional dependencies in opkg like you have in most desktop linux. Think of average case of 8-16-32MB ROM where having that vxlan for swarm would fill the / denying normal default sandbox mode of docker.

What I mean is an interception of the docker swarm command and showing a warning if docker-mod-swarm isn't installed. This would come as part of the docker package, which is not in the default install package list. It could be done via a proxy script for docker.

[brada4]

I suspect there is older docker package at hand, because current OpenWRT package makes conntrack rules and does not require state module. OP missed the version numbers.

[matfax]

It's no older package. I'm currently on r16824+125 0d4a0250df17a9f7b0fff720fa3224b45d5b0841 . This is two weeks old. As has been mentioned in the issue description under OpenWrt version.

The OpenWRT package/init scripts/patches do not utilize the state and ipvs extensions, docker does.

[brada4]

docker --version please, and it is not openwrt core package

[matfax]

docker version provides more information:

Client:
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        100c701
 Built:             Wed Dec 21 02:26:23 2022
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       a89b842
  Built:            Wed Dec 21 02:03:42 2022
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:
 runc:
  Version:          1.1.2
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[brada4]

Rather strange, my DOCKER-INGRESS table consists of ctstate rules with same version.
(besides that i got build date in Feb 2023?)

[matfax]

Without the state extension, I'm missing one DNAT rule for the port mapping in one of two DOCKER-INGRESS chains.

[brada4]

restarting firewall flushes docker rules? Could that be the case?

[ynezz]

(besides that i got build date in Feb 2023?)

@brada4 thanks for the help! So IIUC then you're not able to reproduce this and docker swarm works just fine in OpenWrt 21.02, 22.03 and master?

[brada4]

No, 21.02.5, cannot reproduce -m state part at all.
22 is fw4 and nftables based.

[matfax]

I have not restarted the firewall.

@brada4 You didn't provide any reproducible information, so all I can go with are my own instructions. With these instructions, I've setup a vanilla OpenWRT, no custom build, and followed it through. The errors occur exactly as mentioned, are reproduceable on 22.03 (r20028-43d71ad93e). In fact, I even noticed another issue/necessary requirement, i.e. modprobe ip_vs. I also added the optional vxlan, etc. kmods to the installation set. This makes the instructions valid for a fresh vanilla setup.

The docker files that reference -m state are mentioned in the PR. This means, unless iptables -m state yields on OpenWRT 21, the errors will be reproducible on v21 as well.

[brada4]

I just run yout commands and no error with iptables. I guess firewall reload may trigger on some network change,worst is this event id not logged by default.
You report errors against snapshot right after 21.02.5 and older build of docker. Now some 22? Please open issue against openwrt package with repeater and exact package versions.. It just works error-free for me.
Docker documentation describes kmods used for each extra function, not openwrt issue there.