Unable to use two IPs on one Node

[Sapd]

I think describing it as steps to reproduce is the easiest way:

Prerequisites:

• Have two public IP addresses on a Node. Described now as IP A and IP B

Steps to reproduce:

• Check if IP A and IP B are reachable (just to make sure) ping A ping B
• Create two Source-Engine servers (TF2). TF2 Server 1 gets IP A assigned, the other one IP B.
• Start both servers.

Console TF2 Server 1:

Connection to Steam servers successful.
Public IP is IP A

Console TF2 Server 2:

Connection to Steam servers successful.
Public IP is IP A

The last output was no typo, even when starting Server 2, which was assigned IP B, it shows public IP is IP A. This problem makes it unable to get the second server to show up in the server list, also it doesn't let you connect (the last one could depend on the networking setup however).

Troubleshooting

Typing in netstat -ant into the ssh console, to show bindings shows:

tcp 0 0 A:27015 0.0.0.0:* LISTEN
tcp 0 0 B:27015 0.0.0.0:* LISTEN

So bindings/allocations were correct.

I looked deeper in the problem. Docker uses NAT to port forward ports into the internal docker subnet. However for SNAT it simply puts a MASQUERADE rule into the iptables. So the servers will always use the first server IP as outgoing IP address. Using SNAT would solve the problem.

There were even merge requests on docker to fix such kind of issues, but they were not included: moby/moby#8731

Suggestion

Don't let docker handle the IP Port forwarding configuration. Instead, create custom rules.
E.g. for SNAT (ports should be also specified, left out for easy understanding): iptables -t nat -A POSTROUTING -s DOCKERIPOFGAMESERVER -j SNAT --to ALLOCATIONIP

[schrej]

We certainly do not want to fiddle around with iptables ourselves, it just adds an extra layer of complexity to the daemon and will lead to a huge ton of errors, in the worst case we accidentally lock people out of their servers.
We could potentially allow to opt out of the docker mapping and allow to do it yourself if that's possible.
(This is my opinion, maybe Dane would consider this)
This is probably the nicest issue I've seen here btw., detail wise that is, good job :D

[Sapd]

Opting out of the docker mapping would work. However then there also must be a way to supply custom bash commands after starting a container (could be done in the config of the daemon). However the daemon must also then set environment variables for the script, like the ip of the container, the public ip and the ports which should be forwarded.

This would even solve the issue with the extra layer of complexity. Admins who need such functionality could simply copy the bash script with the iptables from the wiki (or somewhere else). This could be even used for other purposes, but then some more environment variables should be set, like hostname etc.

[DaneEveritt]

The only solution I've found for this outside of advanced networking (which is outside the reasonable scope for this software) is to set the network stack used for containers to host.

This can be done by adding a JSON object in core.json for docker.network.name and setting it to host. You'll want to reboot the daemon and rebuild any existing servers if needed, but that will make the entire network stack available to a container.

The downfall of this solution is that there is no isolated network for containers, however if its just your own servers or trusted friends, it should work fine. Then you can do +ip {{SERVER_IP}} in the start arguments, and generally access everything as if you weren't in a docker container (networking wise).

---

Looking at your suggestion, it might be feasible to do that, it just gets complex to manage those routes via the daemon, and should anything go wrong when adding or removing them, it could be ugly.

[Sapd]

Thanks, I tried it and it works.

In my opinion there should be some kind of way for providing custom rules to make it work without the host network option, as especially for bigger hosts, multiple IPs are important.

[kgns]

you should send this as a suggestion to Docker itself. iirc there are a few of them related to this, open for months (maybe years). just search for "outgoing ip feature docker" in google

[DaneEveritt]

I believe we concluded this was due to both Docker and SRCDS. Unfortunately there is very little that can be done to address this at this time. My understanding is that you can add +net_public_address {{SERVER_IP}} to the start line to solve this for now.

I'll keep it on my radar and continue monitoring for changes to Docker that will make this easier to manage in the future.

[Sapd]

Thank you for your time.

I would still regard it as main docker problem. Because srcds assumes something which also should be that way.

Note that the problem is not easily testable.
Some providers of dedicated servers have MAC-filtering on their switches (popular example in Germany: hetzner). That means, that the IP-Adress of the TCP/UDP packet, must match the MAC-Address provided by the provider for that IP.
As docker uses (in our case with multiple ip addresses) a wrong origin address, the packets will go out on the wrong (virtual) interface, packets will be then dropped by such configured switches.

[DaneEveritt]

Yeah that seems like something that boils down to Docker itself. The only solution I can come up with currently is changing the daemon to use the host network stack which then negates the use of isolated networks.

[Dreae]

For those who stumble upon this issue in the future, I have forked mitar's solution from the issue on the docker repository for NAT configuration and updated it to use the SERVER_IP variable set by Pterodactyl to setup SNAT for game servers.

You can find my repository here

I run it using the following systemd service to start the docker container:

[Unit]
Description=Sets up NAT rules for docker
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=/usr/bin/docker pull quay.io/dreae/docker-external-ip
ExecStart=/usr/bin/docker run --net=host --cap-add=NET_ADMIN --cap-add=NET_RAW --volume /var/run/docker.sock:/var/run/docker.sock --restart=always --name %n quay.io/dreae/docker-external-ip
ExecStop=/usr/bin/docker stop %n
ExecStopPost=/usr/bin/docker rm -f %n
ExecReload=/usr/bin/docker restart %n

[Install]
WantedBy=multi-user.target