Issues with ENABLE_AUTOPAUSE=TRUE in Podman (Rootless)

[GeoDerp]

Describe the problem

While enabling autopause, the script has a hard time attaching to the interface (any) while running the container as rootless.
Im assuming there is a way to create an interface that acts like a bridge with permissions for the container to listen. But I'm not sure how to approach such a thing.

I have also been told that Kubernetes has a pause container. Maybe that could be used to in tandem (share network though an autopause container ) with this container to deliver a similar result?

I'm sure there is a way to fix this, if so it would be nice to have this in the readme, as I seem to not be the only one:
#555 (comment)

Container definition

file pulled using (from docker.io). Ports are set in configs (57571 server. 57573 rcon)

podman  pull itzg/minecraft-server

private network mode

podman run --network=private -e AUTOPAUSE_KNOCK_INTERFACE=tap0 -d -v /mnt/mcMain/:/data:Z  -e TYPE=PAPER  -p 57571:57571 -p 57572:57572 -p 57573:57573 -e EULA=TRUE -e ENABLE_AUTOPAUSE=TRUE -e SPIGET_RESOURCES=18494,28140,32748,76177,85151 -e JVM_DD_OPTS=disable.watchdog:true --name mc itzg/minecraft-server

slirp4netns (default )

podman run -e AUTOPAUSE_KNOCK_INTERFACE=tap0 -d -v /mnt/mcMain/:/data:Z  -e TYPE=PAPER  -p 57571:57571 -p 57572:57572 -p 57573:57573 -e EULA=TRUE -e ENABLE_AUTOPAUSE=TRUE -e SPIGET_RESOURCES=18494,28140,32748,76177,85151 -e JVM_DD_OPTS=disable.watchdog:true --name mc itzg/minecraft-server

host

podman run --network=host   -d -v /mnt/mcMain/:/data:Z  -e TYPE=PAPER  -p 57571:57571 -p 57572:57572 -p 57573:57573 -e EULA=TRUE -e ENABLE_AUTOPAUSE=TRUE -e SPIGET_RESOURCES=18494,28140,32748,76177,85151 -e JVM_DD_OPTS=disable.watchdog:true --name mc itzg/minecraft-server

bridge

podman run --network=bridge   -d -v /mnt/mcMain/:/data:Z  -e TYPE=PAPER  -p 57571:57571 -p 57572:57572 -p 57573:57573 -e EULA=TRUE -e ENABLE_AUTOPAUSE=TRUE -e SPIGET_RESOURCES=18494,28140,32748,76177,85151 -e JVM_DD_OPTS=disable.watchdog:true --name mc itzg/minecraft-server

Container logs

private

$ podman logs mc
[init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x. 1 1000 1000 594 Jan 12 05:10 /data'
[init] Resolved version given LATEST into 1.18.1
[init] Autopause functionality enabled
could not open tap0: tap0: You don't have permission to capture on that device (socket: Operation not permitted)
[Autopause loop] Failed to start knockd daemon.
[Autopause loop] Probable cause: Unable to attach to interface "tap0".
[Autopause loop] Available interfaces within the docker container:
[Autopause loop]   lo tap0
[Autopause loop] Please set the environment variable AUTOPAUSE_KNOCK_INTERFACE to the interface that handles incoming connections.
[Autopause loop] If unsure which interface to choose, run the ifconfig command in the container.
[Autopause loop] Autopause failed to initialize. This log entry will be printed every 30 minutes.
[init] Resolving type given PAPER
[init] Removing old PaperMC versions ...
[init] Downloading PaperMC 1.18.1 (build 150) ...

slirp4netns

$[core@mc-raspi4 ~]$ podman logs mc
[init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x. 1 1000 1000 594 Jan 13 07:03 /data'
[init] Resolved version given LATEST into 1.18.1
[init] Autopause functionality enabled
could not open tap0: tap0: You don't have permission to capture on that device (socket: Operation not permitted)
[Autopause loop] Failed to start knockd daemon.
[Autopause loop] Probable cause: Unable to attach to interface "tap0".
[Autopause loop] Available interfaces within the docker container:
[Autopause loop]   lo tap0
[Autopause loop] Please set the environment variable AUTOPAUSE_KNOCK_INTERFACE to the interface that handles incoming connections.
[Autopause loop] If unsure which interface to choose, run the ifconfig command in the container.
[Autopause loop] Autopause failed to initialize. This log entry will be printed every 30 minutes.

Host

[core@mc-raspi4 ~]$ podman run --network=host   -d -v /mnt/mcMain/:/data:Z  -e TYPE=PAPER  -p 57571:57571 -p 57572:57572 -p 57573:57573 -e EULA=TRUE -e ENABLE_AUTOPAUSE=TRUE -e SPIGET_RESOURCES=18494,28140,32748,76177,85151 -e JVM_DD_OPTS=disable.watchdog:true --name mc itzg/minecraft-server
Port mappings have been discarded as one of the Host, Container, Pod, and None network modes are in use
210cfb0f7f35bbd0dd1562794fd09a5be5ed7199402d83df96e505fc985372ad
[core@mc-raspi4 ~]$ podman logs mc
[init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x. 1 1000 1000 594 Jan 13 07:03 /data'
[init] Resolved version given LATEST into 1.18.1
[init] Autopause functionality enabled
could not open eth0: eth0: You don't have permission to capture on that device (socket: Operation not permitted)
[Autopause loop] Failed to start knockd daemon.
[Autopause loop] Probable cause: Unable to attach to interface "eth0".
[Autopause loop] Available interfaces within the docker container:
[Autopause loop]   eth0 lo wlan0
[Autopause loop] Please set the environment variable AUTOPAUSE_KNOCK_INTERFACE to the interface that handles incoming connections.

bridge

[core@mc-raspi4 ~]$ podman logs mc
[init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x. 1 1000 1000 594 Jan 13 07:03 /data'
[init] Resolved version given LATEST into 1.18.1
[init] Autopause functionality enabled
could not open eth0: eth0: You don't have permission to capture on that device (socket: Operation not permitted)
[Autopause loop] Failed to start knockd daemon.
[Autopause loop] Probable cause: Unable to attach to interface "eth0".
[Autopause loop] Available interfaces within the docker container:
[Autopause loop]   eth0 lo
[Autopause loop] Please set the environment variable AUTOPAUSE_KNOCK_INTERFACE to the interface that handles incoming connections.
[Autopause loop] If unsure which interface to choose, run the ifconfig command in the container.
[Autopause loop] Autopause failed to initialize. This log entry will be printed every 30 minutes.

[itzg]

I'm going to explore a root-less approach when I finally get to this idea itzg/mc-server-runner#19

[dogtopus]

For others who might be looking for workarounds: you might be able to get away with systemd socket activation

[philipwilk]

I'm pretty sure it works if you pass -e AUTOPAUSE_KNOCK_INTERFACE=tap0 and --cap-add=CAP_NET_RAW to podman run.
Well, pausing after no connections seems to work. Either I can't seem to get it to finish pausing or it doesn't like whatever it does after pausing (it stops doing anything after this). I've tried a vanilla, paper and quilt modded server and they all exhibit this.

[jmighion]

I finally got it after nearly 2 years! The solution was stupidly simple too and I'm kicking myself for not figuring this out earlier.

TL;DR run with -e AUTOPAUSE_KNOCK_INTERFACE=tap0 --cap-add=CAP_NET_RAW --network slirp4netns:port_handler=slirp4netns

I was looking back through this old thread and saw this comment about watching the interface while knocking. With a normal rootless setup I was watching with tcpdump -ani tap0 and with bridge networking or running as root I was watching with tcpdump -ani eth0. With root I was able to see the traffic from knock -v <my.ipv4> 25565, but with bridge or normal networking nothing was coming through from the knock client. I could however see the games traffic when connected and not stuck in autopause (open another terminal, exec into the container, and run rcon-cli). So I figured it has to be something related to the ports somehow.

I then looked at the bound ports from the host in rootless and rootfull setups.
Rootless:

[root@nuc ~]# netstat -tulpn | grep LISTEN
...output omitted...
tcp6       0      0 :::25565                :::*                    LISTEN      406372/rootlessport

Rootfull:

[root@nuc ~]# netstat -tulpn | grep LISTEN
...output omitted...
tcp        0      0 0.0.0.0:25565           0.0.0.0:*               LISTEN      399784/conmon

Seeing how rootlessport was always on ipv6, I tried knock -v -6 ::1 25565 and knock -v -6 <my.ipv6> 25565. Still nothing on tcpdump for the normal interfaces. I did see the knock traffic on lo, but the docs for this container suggests to not do that. I did it anyway and got [Autopause loop] Selected interface "lo" does not exist!. It's there, just not available from what mc-image-helper network-interfaces reports back. I can exec in and run /usr/local/sbin/knockd -c /tmp/knockd-config.cfg -d -i lo -D successfully but still can't run the /auto/autopause-daemon.sh script.

The rootlessport was the next clue. Went googlin', was reading through the network options, and found this comment containers/podman#14425 (comment). Saw they mentioned that rootlessport was a port forwarder, but so was slirp4netns (among others).

From the network options and this rootless containers site, I saw the option to set the port forwarder with --network slirp4netns:port_handler=slirp4netns. Now the listening port is on an ipv4 address:

[root@nuc ~]# netstat -tulpn | grep LISTEN
...output omitted...
tcp        0      0 0.0.0.0:25565           0.0.0.0:*               LISTEN      424725/slirp4netns

The rootless container with slirp4netns still has tap0 as the interface and now the tcpdump shows knock traffic there instead of lo. Now it was successfully receiving the knock from the game's multiplayer menu and resuming the game!

The last comment above this one mentioned --cap-add=CAP_NET_RAW which looks like the least privileged option and better than running with --privileged according to the docs.

So nice to have this finally working.

[itzg]

Fantastic write-up @jmighion !

[sbrivio-rh]

The last comment above this one mentioned --cap-add=CAP_NET_RAW which looks like the least privileged option and better than running with --privileged according to the docs.

Actually, since Linux kernel 5.7, commit c427bfec18f2 "net: core: enable SO_BINDTODEVICE for non-root users", unprivileged users (without CAP_NET_RAW) can also use SO_BINDTODEVICE. See this example for how to use this with Podman (you can't use this with the slirp4netns back-end though, you'll need the pasta one).

[jmighion]

Interesting. Have you given that a try to see if autopause works with that?
Looks very new. What benefit would there be going that route vs slirp4netns with CAP_NET_RAW? More restricted privileges it looks like, just not sure at a quick glance.

[sbrivio-rh]

Interesting. Have you given that a try to see if autopause works with that? Looks very new.

No, sorry, actually I'm not familiar at all with this project, I just saw this from a link on containers/podman/issues/14425.

What benefit would there be going that route vs slirp4netns with CAP_NET_RAW? More restricted privileges it looks like, just not sure at a quick glance.

...and, also, I just assumed you used CAP_NET_RAW to bind to a specific interface, but that's not the case — it's because knock uses it, so you would need that anyway.

What the pasta network back-end offers here is simply preserving the correct source address like the "slirp4netns" port forwarding mode does (with the slirp4netns network back-end), while still trying to give you decent throughput. No difference in privileges required in this case.