Debian 12 Docker rootless - modprobe: ERROR - could not insert 'ip_tables': Invalid argument

[Sieboldianus]

Description

I am trying to set up Docker rootless on a fresh Debian 12 following the docs.

Reproduce

I got until:

dockerd-rootless-setuptool.sh install

It prints:

[ERROR] Missing system requirements. Run the following commands to
[ERROR] install the requirements and run this tool again.
[ERROR] Alternatively iptables checks can be disabled with --skip-iptables .

########## BEGIN ##########
sudo sh -eux <<EOF
# Load ip_tables module
modprobe ip_tables
EOF
########## END ##########

When I run the suggested command, I get:

sudo sh -eux <<EOF
# Load ip_tables module
modprobe ip_tables
EOF
+ modprobe ip_tables
modprobe: ERROR: ../libkmod/libkmod-module.c:1047 command_do() Error running install command '/bin/false' for module ip_tables: retcode 1
modprobe: ERROR: could not insert 'ip_tables': Invalid argument

Expected behavior

Docker rootless can be installed in Debian 12.

docker version

Client: Docker Engine - Community
 Version:           27.0.3
 API version:       1.46
 Go version:        go1.21.11
 Git commit:        7d4bcd8
 Built:             Sat Jun 29 00:02:50 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.0.3
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       662f78c
  Built:            Sat Jun 29 00:02:50 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.18
  GitCommit:        ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc:
  Version:          1.7.18
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    27.0.3
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-20-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.58GiB
 Name: ioer-fdz
 ID: 4413cee6-75d9-4200-a074-3bf293337d36
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Installing with skip-iptables works, but then my Docker containers don't have Internet access.

ip_tables is installed and I can successfully use:

sudo iptables -L

Chain INPUT (policy DROP)
target     prot opt source               destination
ufw-before-logging-input  all  --  anywhere             anywhere
ufw-before-input  all  --  anywhere             anywhere
ufw-after-input  all  --  anywhere             anywhere
ufw-after-logging-input  all  --  anywhere             anywhere
ufw-reject-input  all  --  anywhere             anywhere
ufw-track-input  all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
...

[akerouanton]

Thanks for reporting @Sieboldianus.

What's the output of ls -l /etc/alternatives/iptables?

As a side note, I see you have some ufw rules set up. Be aware that both ufw and Docker are currently incompatible, as pointed out in the docs section Docker and ufw.

[Sieboldianus]

$ ls -l /etc/alternatives/iptables
lrwxrwxrwx 1 root root 22 May 12 07:22 /etc/alternatives/iptables -> /usr/sbin/iptables-nft

Thank you regarding the note for ufw - I will see if disabling it solves my problem. Will report back.

[edit]

No change with disabling ufw (sudo ufw disable).

Also tried:

$ sudo sh -eux <<EOF
# Load ip_tables module
modprobe iptables-nft
EOF
+ modprobe iptables-nft
modprobe: FATAL: Module iptables-nft not found in directory /lib/modules/6.1.0-20-amd64
$ sudo sh -eux <<EOF
# Load ip_tables module
modprobe iptables
EOF
+ modprobe iptables
modprobe: FATAL: Module iptables not found in directory /lib/modules/6.1.0-20-amd64

btw. there's also a similar report for Fedora here

[edit2]

Further down the road:

$systemctl enable --now nftables
$systemctl cat nftables.service
# /lib/systemd/system/nftables.service
[Unit]
Description=nftables
Documentation=man:nft(8) http://wiki.nftables.org
Wants=network-pre.target
Before=network-pre.target shutdown.target
Conflicts=shutdown.target
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
StandardInput=null
ProtectSystem=full
ProtectHome=true
ExecStart=/usr/sbin/nft -f /etc/nftables.conf
ExecReload=/usr/sbin/nft -f /etc/nftables.conf
ExecStop=/usr/sbin/nft flush ruleset

[Install]
WantedBy=sysinit.target

# /etc/systemd/system/nftables.service.d/disable-flushing.conf
# DO NOT EDIT! File is managed by Ansible.
[Service]
# don't flush the ruleset when the unit is stopped
ExecStop=

[AkihiroSuda]

Rootless Docker v27.0.3 works fine for me on Debian 12 kernel 6.1.0-22-cloud-amd64.

lsmod, etc.

$ ls -l /etc/alternatives/iptables
lrwxrwxrwx 1 root root 22 Jul  2 04:44 /etc/alternatives/iptables -> /usr/sbin/iptables-nf

$ lsmod
Module                  Size  Used by
xt_nat                 16384  0
xt_tcpudp              20480  0
veth                   36864  0
tun                    61440  3
xt_conntrack           16384  2
nft_chain_nat          16384  6
xt_MASQUERADE          20480  2
nf_nat                 57344  3 xt_nat,nft_chain_nat,xt_MASQUERADE
nf_conntrack_netlink    57344  0
nf_conntrack          188416  5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
xfrm_user              53248  2
xfrm_algo              16384  1 xfrm_user
xt_addrtype            16384  4
nft_compat             20480  8
nf_tables             303104  148 nft_compat,nft_chain_nat
libcrc32c              16384  3 nf_conntrack,nf_nat,nf_tables
nfnetlink              20480  5 nft_compat,nf_conntrack_netlink,nf_tables
br_netfilter           36864  0
bridge                294912  1 br_netfilter
stp                    16384  1 bridge
llc                    16384  2 bridge,stp
overlay               163840  0
isofs                  53248  1
intel_rapl_msr         20480  0
intel_rapl_common      32768  1 intel_rapl_msr
iosf_mbi               24576  1 intel_rapl_common
crct10dif_pclmul       16384  1
crc32_pclmul           16384  0
ghash_clmulni_intel    16384  0
sha512_ssse3           49152  0
sha512_generic         16384  1 sha512_ssse3
sha256_ssse3           36864  0
sha1_ssse3             32768  0
binfmt_misc            24576  1
aesni_intel           393216  0
crypto_simd            16384  1 aesni_intel
nls_ascii              16384  1
cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
nls_cp437              20480  1
mousedev               24576  0
rapl                   20480  0
vfat                   24576  1
fat                    90112  1 vfat
virtio_console         40960  0
virtio_input           16384  0
evdev                  28672  3
button                 24576  0
serio_raw              20480  0
sg                     40960  0
loop                   32768  0
fuse                  176128  1
dm_mod                184320  0
efi_pstore             16384  0
configfs               57344  1
efivarfs               24576  1
qemu_fw_cfg            20480  0
virtio_rng             16384  0
ip_tables              36864  0
x_tables               61440  7 xt_conntrack,nft_compat,xt_tcpudp,xt_addrtype,xt_nat,ip_tables,xt_MASQUERADE
autofs4                53248  2
sr_mod                 28672  1
cdrom                  81920  2 isofs,sr_mod
virtio_scsi            28672  1
virtio_net             73728  0
scsi_mod              274432  3 virtio_scsi,sg,sr_mod
net_failover           24576  1 virtio_net
scsi_common            16384  3 scsi_mod,sg,sr_mod
virtio_blk             28672  2
failover               16384  1 net_failover
virtio_pci             24576  0
virtio_pci_legacy_dev    16384  1 virtio_pci
crc32c_intel           24576  3
virtio_pci_modern_dev    20480  1 virtio_pci
virtio                 20480  7 virtio_rng,virtio_console,virtio_scsi,virtio_input,virtio_pci,virtio_blk,virtio_net
virtio_ring            45056  7 virtio_rng,virtio_console,virtio_scsi,virtio_input,virtio_pci,virtio_blk,virtio_net

[AkihiroSuda]

• Is this some non-standard Debian variant such as WSL2 or something?
• Do you see any error in dmesg?

[Sieboldianus]

Thanks for the sleuthing.

This is a standard dedicated VM in a big cluster (vmware). Never had problems with Docker there and also successfully used rootless Docker, just with Ubuntu.

lsmod

Module                  Size  Used by
xt_conntrack           16384  1
nft_chain_nat          16384  3
xt_MASQUERADE          20480  1
nf_nat                 57344  2 nft_chain_nat,xt_MASQUERADE
nf_conntrack_netlink    57344  0
xfrm_user              53248  1
xfrm_algo              16384  1 xfrm_user
xt_addrtype            16384  2
nft_compat             20480  4
x_tables               61440  4 xt_conntrack,nft_compat,xt_addrtype,xt_MASQUERADE
br_netfilter           36864  0
bridge                311296  1 br_netfilter
stp                    16384  1 bridge
llc                    16384  2 bridge,stp
vsock_loopback         16384  0
vmw_vsock_virtio_transport_common    53248  1 vsock_loopback
vmw_vsock_vmci_transport    36864  1
vsock                  53248  5 vmw_vsock_virtio_transport_common,vsock_loopback,vmw_vsock_vmci_transport
overlay               163840  0
sunrpc                692224  1
binfmt_misc            24576  1
nft_reject_inet        16384  3
nf_reject_ipv4         16384  1 nft_reject_inet
nf_reject_ipv6         20480  1 nft_reject_inet
nft_reject             16384  1 nft_reject_inet
nf_log_syslog          24576  14
nft_log                16384  7
nft_limit              16384  7
intel_rapl_msr         20480  0
intel_rapl_common      32768  1 intel_rapl_msr
ghash_clmulni_intel    16384  0
sha512_ssse3           49152  0
nft_ct                 24576  29
sha512_generic         16384  1 sha512_ssse3
nf_conntrack          188416  5 xt_conntrack,nf_nat,nft_ct,nf_conntrack_netlink,xt_MASQUERADE
sha256_ssse3           32768  0
sha1_ssse3             32768  0
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
vmw_balloon            24576  0
vmwgfx                380928  2
aesni_intel           393216  0
crypto_simd            16384  1 aesni_intel
cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
drm_ttm_helper         16384  1 vmwgfx
ttm                    94208  2 vmwgfx,drm_ttm_helper
pcspkr                 16384  0
drm_kms_helper        208896  1 vmwgfx
vmw_vmci               98304  2 vmw_balloon,vmw_vsock_vmci_transport
sg                     40960  0
joydev                 28672  0
evdev                  28672  2
serio_raw              20480  0
ac                     20480  0
button                 24576  0
nf_tables             303104  360 nft_ct,nft_compat,nft_log,nft_reject_inet,nft_chain_nat,nft_limit,nft_reject
nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
drm                   614400  6 vmwgfx,drm_kms_helper,drm_ttm_helper,ttm
fuse                  176128  1
loop                   32768  0
efi_pstore             16384  0
configfs               57344  1
autofs4                53248  2
btrfs                1789952  2
blake2b_generic        20480  0
xor                    24576  1 btrfs
raid6_pq              122880  1 btrfs
zstd_compress         294912  1 btrfs
libcrc32c              16384  4 nf_conntrack,nf_nat,btrfs,nf_tables
crc32c_generic         16384  0
dm_mod                184320  13
sd_mod                 65536  3
t10_pi                 16384  1 sd_mod
sr_mod                 28672  0
cdrom                  81920  1 sr_mod
crc64_rocksoft         20480  1 t10_pi
crc64                  20480  1 crc64_rocksoft
ata_generic            16384  0
crc_t10dif             20480  1 t10_pi
crct10dif_generic      16384  0
ata_piix               45056  0
libata                401408  2 ata_piix,ata_generic
vmxnet3                73728  0
crct10dif_pclmul       16384  1
crct10dif_common       16384  3 crct10dif_generic,crc_t10dif,crct10dif_pclmul
crc32_pclmul           16384  0
vmw_pvscsi             32768  1
crc32c_intel           24576  3
psmouse               184320  0
scsi_mod              286720  5 vmw_pvscsi,sd_mod,libata,sg,sr_mod
i2c_piix4              28672  0
scsi_common            16384  4 scsi_mod,libata,sg,sr_mod

Nothing unusual in dmesg.

Above with systemctl cat nftables.service I see that some modifications appear to be made by admins:

# /etc/systemd/system/nftables.service.d/disable-flushing.conf
# DO NOT EDIT! File is managed by Ansible.

But I doubt this should be a problem.

[Sieboldianus]

Now I am having the same issue on another VM, this time with Ubuntu 24.04 LTS.

1. Run updates:

sudo apt-get update && sudo apt-get upgrade

2. If there's a kernel update reboot your system before trying sudo iptables -L again.

3. sudo iptables -L

> Chain INPUT (policy ACCEPT)
> target     prot opt source               destination

> Chain FORWARD (policy ACCEPT)
> target     prot opt source               destination

> Chain OUTPUT (policy ACCEPT)
> target     prot opt source               destination

Check that iptables is built into the kernel:

grep -i iptables /boot/config-$(uname -r)
> CONFIG_IP_NF_IPTABLES=m
> CONFIG_IP6_NF_IPTABLES=m
> # iptables trigger is under Netfilter config (LED target)

If the output references something like *IPTABLES=y then it's already built in (?)

Make sure the module is loaded:

sudo modprobe nf_tables
sudo modprobe ip_tables

modprobe: ERROR: ../libkmod/libkmod-module.c:1084 command_do() Error running install command '/bin/false' for module ip_tables: retcode 1
modprobe: ERROR: could not insert 'ip_tables': Invalid argument

modinfo ip_tables
filename:       /lib/modules/6.8.0-36-generic/kernel/net/ipv4/netfilter/ip_tables.ko.zst
description:    IPv4 packet filter
> author:         Netfilter Core Team <coreteam@netfilter.org>
> license:        GPL
> srcversion:     48318B01494EA0C1C2A27E9
> depends:        x_tables
> retpoline:      Y
> intree:         Y
> name:           ip_tables
> vermagic:       6.8.0-36-generic SMP preempt mod_unload modversions
> sig_id:         PKCS#7
> signer:         Build time autogenerated kernel key
> sig_key:        6A:DF:2A:58:A0:7C:DC:84:51:A8:A1:47:65:C5:D3:08:29:3E:E8:67
> sig_hashalgo:   sha512
> signature:      64:E8:11:CF:A7:FA:8C:FC:1B:D4:46:BE:8D:2A:CA:AD:BB:9B:37:A9:
>                 66:46:84:F1:25:2A:8C:E3:D7:87:59:23:7C:7F:11:2A:7E:38:DB:F9:
>                 24:FA:FF:A6:BE:39:8D:98:6A:AF:A6:33:CA:49:D8:E9:C8:5C:1F:2E:
>                 94:7F:2B:C1:63:FC:B8:2E:9D:3A:CB:D8:BB:FE:51:7D:F9:F9:97:D9:
>                 DA:63:81:BC:E3:21:71:B4:58:4C:D6:C7:A0:4D:EB:79:48:9F:DA:9C:
>                 8E:DB:21:17:F9:64:72:75:31:5E:B6:25:E8:66:68:E5:B3:20:4B:66:
>                 38:97:0B:1E:9F:13:B9:CE:1F:1C:54:E7:E2:41:DD:22:09:67:82:15:
>                 A1:0F:28:43:6F:71:93:E6:7E:7C:2E:D4:71:3C:E8:B5:E1:A0:34:98:
>                 C8:3F:48:20:59:FD:EC:2F:88:D2:C6:3D:4A:61:33:54:8E:F0:E8:37:
>                 32:36:9F:B9:DA:1C:AC:A0:C9:DB:E2:BF:CC:B7:42:1B:00:AA:43:DB:
>                 F5:B0:D2:52:C8:08:30:34:89:FE:15:F0:AC:B0:FD:C4:B6:6C:24:DD:
>                 DD:EA:F5:54:DB:6F:31:7A:93:1E:E0:C2:EB:88:9B:07:63:DB:FC:D5:
>                 56:35:31:BE:1A:98:C1:AA:06:1F:95:D1:B7:75:F6:DA:E2:FD:F4:BB:
>                 1A:30:2D:D6:EE:E3:00:64:43:0B:8C:8D:DB:0A:87:A9:C0:89:9E:31:
>                 25:8D:FC:57:AB:59:BF:19:58:C4:6B:9D:EA:A4:4A:44:02:21:87:16:
>                 AE:6C:EF:AF:D6:7A:EA:DC:08:7C:A3:1C:E7:28:D5:12:98:20:BF:02:
>                 74:1C:95:15:8A:EE:60:1D:01:CC:1E:88:0C:C5:2C:9D:3F:9B:1D:B0:
>                 96:31:27:B5:2D:FF:29:A6:83:0D:62:E1:6D:AF:72:E2:BE:B3:A7:0E:
>                 AA:DE:EA:A7:8B:47:F5:38:C8:A9:C2:94:AE:4C:93:5A:2D:BA:91:C7:
>                 A8:D9:00:5A:79:A6:B9:96:69:32:CD:9D:4D:24:31:0E:FA:7C:9C:A5:
>                 0C:E1:2B:F2:37:AF:20:24:3E:08:5A:00:3B:07:DF:66:1D:DC:2F:EF:
>                 F9:1F:57:24:DB:BC:D6:21:73:4F:22:9F:1F:98:4B:00:36:34:6A:68:
>                 28:1D:7A:93:A5:E2:39:69:FB:26:1D:3A:47:8C:93:55:A3:55:BD:15:
>                 22:B9:BE:5C:23:3F:F0:5A:C5:D5:9F:8D:0E:E5:07:DA:F5:A4:02:47:
>                 5B:D9:0C:4A:B8:C4:6A:2F:0E:73:99:A7:F7:74:FA:CD:5E:B6:1D:28:
>                 5E:41:D2:CA:F3:B9:60:BA:6F:95:5A:C1

[Sieboldianus]

Found it! @AkihiroSuda @akerouanton

I had to use:

insmod /lib/modules/6.8.0-36-generic/kernel/net/ipv4/netfilter/ip_tables.ko.zst
# or
# insmod /lib/modules/`uname -r`/kernel/net/ipv4/netfilter/ip_tables.ko.zst

Afterwards, the following worked:

sudo sh -eux <<EOF
# Load ip_tables module
modprobe ip_tables
EOF
+ modprobe ip_tables

I found this by looking at the output of modinfo ip_tables, which pointed to nf_tables.ko.zst (instead of nf_tables.ko) (Ubuntu 24.0.4. Under Debian 12, the name was nf_tables.ko, but it could equally be loaded with insmod - instead of modprobe, where the error occurred.

I added my experiences to this blog post: https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless/

[Sieboldianus]

I am leaving this open, in case you want to update the hint that is shown.