Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
GitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
HAProxy does not respond after update to 1409.2.0 #2022
Container Linux Version
AWS t1.small instance HVM
Marathon-LB HAProxy should respond on port
This is inside Docker container, but same applies when accessing the port from outside.
Marathon-LB was happily running on a host, but today, after update to 1409.2.0 it stopped responding on the port it binds to.
All the requests hang as if they are being firewalled. All other ports are accessible. Tried changing port to a different number, but it still hangs. Works fine on previous OS versions, happen immediately after update.
I'm able to reproduce this with the following and comparing v1353.8.0 to v1409.5.0
$ docker run -p 2181:2181 --restart always -d zookeeper@sha256:6308fff92245ff7232e90046976d2c17ffb363ae88c0d6208866ae0ab5a4b886 $ docker run --privileged -e MESOS_WORK_DIR=/tmp --net=host -d mesosphere/marathon@sha256:55a0d07ab9182e0908d3256435679eede6158b6a0ac956d048c151ffcd8eee32 --master=local --zk zk://127.0.0.1:2181/marathon $ docker run -d --net=host --privileged -it -e PORTS=9090 mesosphere/marathon-lb@sha256:563d84e8d1444f68d13f03be48d39ec0eb7d5bbaab0b4c26ba9b905de4009900 sse --group external --marathon http://127.0.0.1:8080 $ sleep 5 $ curl localhost:9090 # Either hangs or errors depending on the version
Notably, iptables rules post-update have the following rules which aren't present on the older version (1353.8.0):
These rules, I think, are being added by marathon-lb's startup script. See https://github.com/mesosphere/marathon-lb/blob/b950d727be15be1e467fd56c458a015e907d861e/service/haproxy/run#L5-L17
There might be some sort of kernel change/regression causing this.
Note with the host iptables version (v1.4.21) I can't reproduce this, but using a more recent iptables (e.g. one in fedora:25) I can reproduce this reliably with something like:
I believe this is the same as https://bugzilla.redhat.com/show_bug.cgi?id=1459676, where a kernel patch is referenced, which is already included in an upstream point release it looks like.
As one fortunate data-point, some tools (such as kubernetes/kube-proxy) are packaged with old enough iptables versions that this doesn't impact them at the moment, and things on the host (e.g. docker itself) which exec iptables directly aren't impacted I think.