nspawn ports binding excludes loopback

[fsateler]

Submission type

• Bug report

systemd version the issue has been seen with

232

NOTE: Do not submit bug reports about anything but the two most recently released systemd versions upstream!

Used distribution

Debian

In case of bug report: Expected behaviour you didn't see

Starting a nspawn container with the -p$port option allows one to connect to localhost:$port

In case of bug report: Unexpected behaviour you saw

Connecting to $public_ip:$port works, but connecting to localhost:$port doesn't. This is the iptables nat config nspawn generated for my container:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp  --  anywhere             anywhere             tcp dpt:7689 ADDRTYPE match dst-type LOCAL to:10.0.0.7:7689
<snip>
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp  --  anywhere            !loopback/8           tcp dpt:7689 ADDRTYPE match dst-type LOCAL to:10.0.0.7:7689

In case of bug report: Steps to reproduce the problem

Create a container with a network-listening service, and add a nspawn file exposing that port. Connections will be possible to the public ip address, but not to localhost.

[parmentelat]

I am experiencing similar issues; for the record I have posted this matter also on SO at https://stackoverflow.com/questions/44338686/how-use-systemd-nspawn-with-network-veth-and-port-n-and-p/44602835#44602835

systemd version the issue has been seen with

# rpm -q systemd
systemd-231-15.fc25.x86_64

Used distribution

# cat /etc/fedora-release
Fedora release 25 (Twenty Five)

In case of bug report: Expected behaviour you didn't see

I would expect to use --private-network -p 10000:20000 and then having localhost:10000 on the host side be wired into localhost:20000 on the container side

In case of bug report: Unexpected behaviour you saw

port 10000 is not reachable on the host side

In case of bug report: Steps to reproduce the problem

machinectl pull-raw --verify=no https://download.fedoraproject.org/pub/fedora/linux/releases/25/CloudImages/x86_64/images/Fedora-Cloud-Base-25-1.3.x86_64.raw.xz
systemd-nspawn -M Fedora-Cloud-Base-25-1.3.x86_64.raw --private-network -p 10000:20000 nc -l localhost 20000

And then in another terminal

echo hello | nc localhost 10000
Ncat: Connection refused.

Additional material

Having read the manual page for systemd-nspawn rather thoroughly, I would appreciate, regardless of this issue, some more details about the expected interactions that @fsateler suggests systemd-nspawn has with iptables

I am playing with this on an alpha-production box that runs a docker-based system already, and so it does mess with iptables on its own; so it would help to assess if running both contenairization tools concurrently is likely to create any conflict.

[parmentelat]

As a follow-up on my last comment, I stumbled on this link
https://lists.freedesktop.org/archives/systemd-devel/2015-May/032531.html

that suggests that systemd does not use iptables anymore (taking into account that this announcement was made 2 years ago), but it is unclear to me if/how iptables as used by docker coexist with nftables as used by systemd

I feel like I am missing something huge, any clarification welcome

[lahwaacz]

Fixing this would be easy - just avoid these lines:

systemd/src/shared/firewall-util.c

Lines 301 to 306 in a859abf

	/* Don't apply onto loopback addresses */
	if (!destination) {
	entry->ip.dst.s_addr = htobe32(0x7F000000);
	entry->ip.dmsk.s_addr = htobe32(0xFF000000);
	entry->ip.invflags = IPT_INV_DSTIP;
	}

The question is if they should be just deleted or disabled via a new parameter for use by nspawn. The exclusion was probably added for a reason...

[dylrich]

Just wanted to add here that adding an option for nspawn to work with the loopback interface would be much appreciated here!

[drokmed]

6 years later, nothing? really?

[martinpitt]

Is --port 10022:22 even supposed to work that way? I.e. is this a supported use case? I've tried it with --private-network and --network-veth, on Ubuntu 22.04, RHEL 10, and Fedora 40. In Ubuntu 22.04, ssh'ing to hostpublicIP:10022 gets to the host, i.e. it redirects port 10022 to the host port 22 (not the container). In RHEL 10 and Fedora, port 10022 refuses connection on any IP (localhost or public).

I am trying to find a way to get to the container's ssh across a wide range of distros (for CI). --network-veth works fine with networkd, but RHEL 10 and Fedora CoreOS don't have networkd. I was hoping to avoid all the network/bridge/NetworkManager/DHCP/etc. stuff and just deal with ssh port redirection 😢

[andykais]

Just adding to this discussion, if --private-network + --port is not a supported workflow for local web apps, what is the suggested set up for a container with no internet access that exposes a port to localhost? This is basically how I want to test out just about any web app that I download from the internet.