This repository has been archived by the owner on Dec 5, 2017. It is now read-only.
cannot connect to Docker-exposed port of same host in elastic Mesos (e.g. GCE, digitalocean, ...) #94
Comments
|
Firewall configuration on the GCE slave: jclouds@development-2451-650:~$ sudo iptables-save
# Generated by iptables-save v1.4.14 on Tue Dec 2 21:53:25 2014
*nat
:PREROUTING ACCEPT [3962:236852]
:INPUT ACCEPT [3635:217232]
:OUTPUT ACCEPT [594:39206]
:POSTROUTING ACCEPT [605:39866]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 31020 -j DNAT --to-destination 172.17.0.2:6379
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 31030 -j DNAT --to-destination 172.17.0.4:80
COMMIT
# Completed on Tue Dec 2 21:53:25 2014
# Generated by iptables-save v1.4.14 on Tue Dec 2 21:53:25 2014
*filter
:INPUT DROP [488:29280]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [150:10397]
-A INPUT -s 10.181.4.180/32 -j ACCEPT
-A INPUT -s 10.99.68.149/32 -j ACCEPT
-A INPUT -s 10.158.11.199/32 -j ACCEPT
-A INPUT -s 10.134.57.34/32 -j ACCEPT
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m state --state NEW,ESTABLISHED -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 1194 -j ACCEPT
-A INPUT -i tun0 -j ACCEPT
-A INPUT -i lo -m comment --comment "Allow loopback" -j ACCEPT
-A INPUT -p icmp -m comment --comment "Allow ping" -j ACCEPT
-A FORWARD -d 172.17.0.4/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A FORWARD -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 6379 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -o eth0 -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT
COMMIT
# Completed on Tue Dec 2 21:53:25 2014 |
|
Docker info: jclouds@development-2451-650:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4076df02c732 jdef/php-redis:latest "/bin/sh -c /run.sh" About an hour ago Up About an hour k8s_php-redis.5fa73094_e8f87f0f-7a65-11e4-8b8c-42010a863922.mesos_e8f87f0f-7a65-11e4-8b8c-42010a863922_74bd198e
0bdd86a7ea56 kubernetes/pause:go "/pause" About an hour ago Up About an hour 0.0.0.0:31030->80/tcp k8s_net.62283c5_e8f87f0f-7a65-11e4-8b8c-42010a863922.mesos_e8f87f0f-7a65-11e4-8b8c-42010a863922_bd597ef8
32d5b8586ff1 jdef/redis-slave:latest "/bin/sh -c /run.sh" About an hour ago Up About an hour k8s_slave.a9a11e43_bda7385a-7a61-11e4-8b8c-42010a863922.mesos_bda7385a-7a61-11e4-8b8c-42010a863922_81074f7d
6fc1da2e00a3 kubernetes/pause:go "/pause" About an hour ago Up About an hour 0.0.0.0:31020->6379/tcp k8s_net.d8aa8435_bda7385a-7a61-11e4-8b8c-42010a863922.mesos_bda7385a-7a61-11e4-8b8c-42010a863922_985a37f3
jclouds@development-2451-650:~$ sudo docker version
Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa
OS/Arch (client): linux/amd64
Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 39fa2fa |
|
Docker processes: jclouds@development-2451-650:~$ ps auxwww|grep -e docker|grep -v -e grep
root 2027 1.9 0.3 1158420 23536 ? Sl 19:59 2:29 /usr/bin/docker -d -p /var/run/docker.pid
root 2383 2.2 0.7 389748 57108 ? Sl 19:59 2:50 /usr/local/sbin/mesos-slave --master=zk://10.134.57.34:2181/mesos --log_dir=/var/log/mesos --containerizers=docker,mesos --executor_registration_timeout=5mins --hostname=10.181.4.180 --ip=10.181.4.180 --attributes=host:development-2451-650.c.k8s-mesos.internal
root 3343 0.0 0.1 247444 11332 ? Sl 20:28 0:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 31020 -container-ip 172.17.0.2 -container-port 6379
root 4505 0.0 0.1 198428 11540 ? Sl 20:58 0:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 31030 -container-ip 172.17.0.4 -container-port 80 |
jdef
changed the title
|
xref: moby/moby#6810 For some reason this was reverted after it was merged to master: |
|
This breaks the actual usability of the guestbook. While all the containers are, in fact running, and the HTTP server will happily serve up a web page - it does NOT actually connect to redis, and instead times out. |
|
cc @cmaloney for more ideas on this. |
|
Playing around with a docker-1.3.1-dev build that has the hairpin change installed. Also had to add some iptables rules to get connectivity to work from within the frontendController container to the :; iptables -A INPUT -i eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
:; iptables -A INPUT -p tcp -m state --state NEW,ESTABLISHED -m tcp --dport 10000 -j ACCEPT
:; iptables -A OUTPUT -o eth1 -j ACCEPT
:; iptables -A OUTPUT -p tcp -m tcp --sport 10000 -m state --state ESTABLISHED -j ACCEPTredis-cli (executed from within the frontendController) is still experiencing Connection Reset errors. tcpdump looks like this: current iptables: |
|
FWIW, my browser currently returns this: <br />
<b>Fatal error</b>: Uncaught exception 'Predis\Connection\ConnectionException' with message 'Error while reading line from the server [tcp://10.132.189.240:10000]' in /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php:141
Stack trace:
#0 /vendor/predis/predis/lib/Predis/Connection/StreamConnection.php(208): Predis\Connection\AbstractConnection->onConnectionError('Error while rea...')
#1 /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php(130): Predis\Connection\StreamConnection->read()
#2 /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php(122): Predis\Connection\AbstractConnection->readResponse(Object(Predis\Command\StringGet))
#3 /vendor/predis/predis/lib/Predis/Client.php(246): Predis\Connection\AbstractConnection->executeCommand(Object(Predis\Command\StringGet))
#4 /vendor/predis/predis/lib/Predis/Client.php(228): Predis\Client->executeCommand(Object(Predis\Command\StringGet))
#5 /app/index.php(32): Predis\Client->__call('get', Array)
#6 /app/index.php(32): Predis\Client->get('me in <b>/vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php</b> on line <b>141</b><br /> |
|
from redis-cli within frontendController: |
|
xref #93 |
|
Feeling pretty silly - the It's not immediately clear to me why the iptables rule below is needed. Perhaps the upstream rebase to 0.5x+ (see #88) will obviate the need for the docker hairpin networking? |
|
Data point: with iptables rules: root@development-1823-93f:~/docker# iptables-save|grep -e 10000
-A INPUT -p tcp -m state --state ESTABLISHED -m tcp --dport 10000 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 10000 -j LOG
-A INPUT -p tcp -m state --state NEW -m tcp --dport 10000 -j ACCEPT |
|
The default rule for INPUT is currently DROP. We should probably allow all packets from the docker0 bridge to the host. Perhaps the default elastic mesosphere cluster configuration could take this into account. Note: with these additional rules, the hacky docker hairpin networking is not required for the guestbook to work properly. |
|
mesosphere has made changes to the default firewall rules in GCE, as per the prior comment: works like a champ! In order to access the frontend-service form the internet, two remaining steps are needed (reasonable):
$ iptables -A INPUT -i eth0 -p tcp -m state --state NEW,ESTABLISHED -m tcp --dport 9998 -j ACCEPT
$ iptables -A OUTPUT -p tcp -m tcp --sport 9998 -m state --state ESTABLISHED -j ACCEPTGCE rule: |
|
via michael jin (mesosphere): OK the IPTable changes have made their way to DO as well. Let us know if you run into anything else! |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
[[EDIT]]
How I got here:
Need to reproduce elsewhere...
The text was updated successfully, but these errors were encountered: