Docker changes source IP address from incoming external UDP packets

[gabrielmocan]

Dear colleagues,

I'm currently dockerizing a traffic analyzer application which I developed in Python and it's already being used in production. I'm almost there, the first step I took was to create the image of my app based on Alpine 3.6. The main dependency for my code to run is the nfdump project (available at https://github.com/phaag/nfdump). This project delivers a collector for Netflow/IPFIX UDP packets called "nfcapd", which runs on port 9995 (properly exposed on my Dockerfile).

The thing is, If I run the container with default network and publishing port 9995, the UDP packets coming from different routers (with different IPs, obviously) all come with the same docker gateway IP (172.18.0.1) -- but I need original source IP address in order to determine from which router the packets belongs to.

So I searched for solutions and stumbled with the --net=host option. If I run the container with run command and passing --net=host, the source IP of the incoming UDP packets is not altered, solving my problem aparently. But then again, the thing is, this container is not running alone in the service. Inside the docker-compose.yml I've created some other services, such as db (InfluxDB) which has to be reachable by the collector container (it collects the packets, process them, and send the results to InfluxDB).

Wrapping it up, If I use the host option for the collector container network, it receives the correct source IP addresses, but in the other hand, it cant communicate to the other containers which runs in the stack. The DNS does not work and I cant resolve the db container name from within the collector container. By obvious reasons I cant rely on IPs, as taught by you, professor, as I expect to run multiple instances of this service to serve different clients from a single machine (swarm is not a requirement in this project).

[jensenja]

Hi,

While I realize it probably doesn't help, I ran into the same problems when I tried to run a Logstash Docker container to receive UDP syslog from my network gear. Some of my source IP's were being changed to the Docker bridge IP address - others were not. This appears to be a known issue and many github issues have been filed for it, with no clear resolution to any of them:

moby/moby#18845
moby/moby#16720
moby/moby#7540
hardware/mailserver#43
moby/moby#15086

I filed my own issue at docker/for-linux#182 before spending hours digging up the previously mentioned ones.

This also appears to maybe impact TCP flows in addition to UDP. I tried both completely disabling IPv6 in my kernel as well as disabling the userland proxy in /etc/docker/daemon.json but neither worked. I eventually just gave up and ran Logstash as a system software package, much to my dismay.

Sorry for the bad news.

[fcrisciani]

@gabrielmocan
@jensenja
It really depends on how you deploy your service, if you use service mesh and expose a port at cluster level so that any node of your cluster can redirect the traffic to the container then yes that is the normal functioning condition due to the use of IPVS as a load balancer front end.

Your options are the following:

1. expose ports on the host (--net=host) and front end your ingress traffic with an external load balancer that will redirect the packet to the host where the container is sitting
2. create a front end global service still with network host and if it has to reach the backend in some way, the backend service will have to expose a port on the ingress and the front end will be able to reach it as localhost:

Hope this clarify

[jensenja]

Hi @fcrisciani

I'm not sure how different @gabrielmocan 's setup is to mine, but I'm only running the docker container service on a single host. And logstash is (was) basically the only container. I'm not running a cluster, not using Swarm, or anything else like that. Again, just a single baremetal server with Docker being the only thing running on it. My UDP traffic sources are all external to the container host - and preserving the source IP of the UDP syslog traffic is obviously important.

Re: options:

1. My particular host has an Internet-facing interface, so using --net=host is undesirable, as I don't believe there's a way to limit --net=host to a single interface. Sure, I could run iptables, but that's yet another stack that I'd like to avoid maintaining/mucking with. I don't have hardware/another system to use as a load balancer, so that's not an option either.

2. I'm not entirely sure what you're getting at with this - a diagram might help.

Thanks for the reply.

[fcrisciani]

@jensenja when you do --net=host you are in the host namespace, so the decision on which interface to listen to is on the application, just don't let it listens on 0.0.0.0 but pass the specific interface

[jensenja]

@fcrisciani of course that would make perfect sense - I'd been struggling with the issue for over 18 hours yesterday when I filed the bug report and I didn't even think to try that out. 🙈

I'll stop derailing @gabrielmocan 's issue at this point. Thanks for the pointer.

[jensenja]

(@fcrisciani if I may - I do have one last question about the behavior)

To give a little more context, there was basically a single offending device that Docker was concealing the source IP for.

Why is it that when I configured the offending device to send syslog to UDP/5014 and then ran the Logstash container with -p 5014:5014/udp the source IP was preserved, but when I changed the device config back to sending syslog on the standard UDP/514 to the container host, and then ran the container with -p 514:514/udp --user=root the source IP was not preserved? (edit: in both cases the application was listening on 0.0.0.0)

[gabrielmocan]

Guys, I appreciate your help but it comes to that I've solved this problem by putting the collector into host network and shared a volume with data processor container, this container though is on normal bridge/overlay network, thus reaching database container normally in order to send processed data to it.

[fcrisciani]

@jensenja not sure I'm following all the details, can you post the 2 different way you are running the container?
(my suggestion of not putting 0.0.0.0 was to address your concern to not have the service exposing the port on the public interface, should not play any other role over here)

[jensenja]

@fcrisciani a more detailed explanation can be found at the github issue I filed (then subsequently closed) with docker-ce for linux (docker/for-linux#182)

[sergey-safarov]

Looks that is root of issue
#2423

[Posnakidesd]

This is old thread nonetheless still relevant.
conntrack -D -p udp after the container has started fixes the issue
moby/moby#16720

[t3chn0m4g3]

Is conntrack -D -p udp still the only solution possible?