Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
316 lines (218 sloc) 9.51 KB

High level Picture

docker network

Testing environment

CentOS (6.5) + Docker (0.7.2)

Refer to http://docs.docker.io/en/latest/examples/running_redis_service/ to create a redis image.

$ sudo docker run -name myredis -d -p 0.0.0.0:40000:6379 andl/redis 
$ sudo docker inspect myredis | grep IPAddress
    "IPAddress”:  “172.17.0.5"

Docker also listen on port 40000, will explain later.

$ sudo netstat -tulpn | grep 40000
tcp        0      0 :::40000          0.0.0.0:*                   LISTEN      1386/docker 

The host network config, you can see the bridge interface (docker0) and virtual eth interface (veth72x2M5).

$ifconfig
docker0   Link encap:Ethernet  HWaddr FE:BB:CE:6D:95:40
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::2438:41ff:fed3:cd69/64 Scope:Link
          ....

eth0      Link encap:Ethernet  HWaddr 00:50:56:A6:99:F1
          inet addr:10.146.22.21  Bcast:10.146.22.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fea6:99f1/64 Scope:Link
          ....

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          ....

veth72x2M5 Link encap:Ethernet  HWaddr FE:BB:CE:6D:95:40
          inet6 addr: fe80::fcbb:ceff:fe6d:9540/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          .... 

The routing table:

[root@wdc-dev ~]# $ ip route
ip route
10.146.22.0/24 dev eth0  proto kernel  scope link  src 10.146.22.21
169.254.0.0/16 dev eth0  scope link  metric 1002
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1
default via 10.146.22.125 dev eth0 

The iptable rules generated by docker as following:

$iptables-save
# Generated by iptables-save v1.4.7 on Sun Jan 26 19:05:26 2014
*raw
:PREROUTING ACCEPT [631:39420]
:OUTPUT ACCEPT [355:92542]
-A PREROUTING -p tcp -m tcp --dport 40000 -j TRACE
-A PREROUTING -p tcp -m tcp --sport 6379 -j TRACE

COMMIT
# Completed on Sun Jan 26 19:05:26 2014
# Generated by iptables-save v1.4.7 on Sun Jan 26 19:05:26 2014
*filter
:INPUT ACCEPT [34819:12573055]
:FORWARD ACCEPT [20:1440]
:OUTPUT ACCEPT [20020:6145857]
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT  
COMMIT
# Completed on Sun Jan 26 19:05:26 2014
# Generated by iptables-save v1.4.7 on Sun Jan 26 19:05:26 2014
*nat
:PREROUTING ACCEPT [13:936]
:POSTROUTING ACCEPT [445:28430]
:OUTPUT ACCEPT [428:27206]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER                                            
-A POSTROUTING -s 172.17.0.0/16 ! -d 172.17.0.0/16 -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -d 10.146.22.21/32 ! -i docker0 -p tcp -m tcp --dport 40000 -j DNAT --to-destination 172.17.0.5:6379     
COMMIT
# Completed on Sun Jan 26 19:05:26 2014

Case 1 (Packet from remote host)

In my dev box(10.110.124.185), I want to access the redis service on 10.146.22.21

$ redis-cli -h 10.146.22.21 -p 40000 GET "docker"
"awesome"

###The request packet flow

  1. In=eth0, SRC=10.110.124.185 DST=10.146.22.21, DPT=40000

    nat:prerouting Matches: A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER

  2. In=eth0, SRC=10.110.124.185 DST=10.146.22.21, DPT=40000

    docker Matches: -A DOCKER -d 10.146.22.21/32 ! -i docker0 -p tcp -m tcp --dport 40000 -j DNAT --to-destination 172.17.0.5:6379

    DNAT: modify DST and DPT

  3. In=eth0 SRC=10.110.124.185 DST=172.17.0.5, DPT=6379

    Routing decision: forward to docker0 interface (according to routing table).

  4. In=eth0, out=docker0, SRC=10.110.124.185 DST=172.17.0.5, DPT=6379

    filter:forward Matches: default policy (ACCEPT)

    nat: postrouting Matches: default policy(ACCEPT)

    To the wire (actually docker0 bridge device)

  5. After find the mac address of 172.17.0.5 using ARP, the bridge forward packet to interface veth72x2M5 since the eth0 of myredis container is connected to that port.

  6. In=eth0, SRC=10.110.124.185 DST=172.17.0.5, DPT=6379

    Frame from eatables: IN=veth72x2M5 OUT= MAC source = 9e:93:85:3f:24:b5 MAC dest = fe:bb:ce:6d:95:40 proto = 0x0806

    ebtables:broute:brouting Matches: default policy ACCEPT (or bridge)

  7. In=eth0, SRC=10.110.124.185 DST=172.17.0.5, DPT=6379

    Please note that this hook was evoked in link layer rather than ip layer.

    nat:prerouing Matches: default policy (ACCPET) since it’s not a local IP (172.17.0.5)

  8. In=eth0, SRC=10.110.124.185 DST=172.17.0.5, DPT=6379

    Bridge decision: to IP protocol:

    If we show mac address on the bridge device, we can see that the dest mac is a ports on bridge, so the bridge decision should send the frame to IP layer.

$ brctl showmacs docker0
  port no mac addr                is local?       ageing timer
  1     fe:bb:ce:6d:95:40       yes                0.00 
  1. In=eth0, SRC=10.110.124.185 DST=172.17.0.5, DPT=6379

Please note that this hook was evoked in eatables.

eatables:filter:input Match: default policy (ACCPET)

  1. In=eth0, SRC=10.110.124.185 DST=172.17.0.5, DPT=6379

    routing decision: Since 172.17.0.5 is the IP for current interface (veth72x2M5), so go to INPUT chain. Our redis process is listen on the interface, so the packet will be send to application.

    The request traversal flow has finished.

The response packet flow

Note that the bridge flow is not mentioned for simplify.

  1. In=docker0 OUT= SRC=172.17.0.5 DST=10.110.124.185 SPT=6379

    conntrack table mark this connection state from NEW to ESTABLISHED since the double direction packets has been seen by firewall.

    nats: prerouting is not consulted to RESTABLISHED connection.

  2. In=docker0 OUT=eth0 SRC=172.17.0.5 DST=10.110.124.185 SPT=6379

    Routing, the packet is routing to eth0 (the default gw).

    filter: forward MATCHES: -A FORWARD -i docker0 ! -o docker0 -j ACCEPT

  3. In=docker0 OUT=eth0 SRC=172.17.0.5 DST=10.110.124.185 SPT=6379

    nat: POSTROUTING MATCHES: -A POSTROUTING -s 172.17.0.0/16 ! -d 172.17.0.0/16 -j MASQUERADE

  4. In=docker0 OUT=eth0 SRC=10.110.124.185 DST=10.110.124.185 SPT=6379

    To the wire.

Case 2 (local access)

You may wonder why docker process also listen on 40000 port. It turns out to help access the service from localhost without add additional SNAT.

$redis-cll -h localhost -p 40000 GET "docker"
"awesome"
  1. In= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 DPT=40000

nat:OUTPUT Matches: default policy (ACCEPT)

filter:OUTPUT

Matches: default policy (ACCEPT)

nat: postrouting Matches: default policy (ACCEPT)

Docker doesn’t have any special rules for packets comes outof lo interface. To the lo device

  1. In=lo Out= SRC=127.0.0.1 DST=127.0.0.1 DPT=40000

    lo receives the packets. Please note conntrack mark the connection as ESTABLISHED. So again, the nat table is not consulted.

  2. Routing, the packets will be processed by application.

filter:input Matches: default policy (ACCEPT)

Docker create a TCP Porxy (proxy/tcp_proxy.go) which listen on 40000 to forward the request to port 7379 on host 172.17.0.5. By forwarding traffic by application layer, docker simplify the iptables rules for local access.

The localhost request traversal is finished.

  1. The foward and response workflow is omitted since no special iptables rules were applied.

Case 3 (Inter container access)

By default, docker enable inter container communication (icc), (refer to rules: -A FORWARD -i docker0 -o docker0 -j ACCEPT) If disable icc using icc=false startup paras, this rule becomes -A FORWARD -i docker0 -o docker0 -j DROP which enhanced the security.

I can create a container with docker “link” function even if icc was disabled.

$ sudo docker run -t -i -entrypoint="/bin/bash" -link myredis:db andl/redis

Docker simply Add two rules to allow icc between sepcific containers:

-A FORWARD -s 172.17.0.2/32 -d 172.17.0.4/32 -i docker0 -o docker0 -p tcp -m tcp --sport 6379 -j ACCEPT
-A FORWARD -s 172.17.0.4/32 -d 172.17.0.2/32 -i docker0 -o docker0 -p tcp -m tcp --dport 6379 -j ACCEPT 

You can access the linked container using ENV var.

root@17ea9bd28f52:/# env
HOSTNAME=17ea9bd28f52
DB_NAME=/tender_poincare/db
DB_PORT_6379_TCP_PORT=6379
TERM=xterm
DB_PORT=tcp://172.17.0.8:6379
DB_PORT_6379_TCP=tcp://172.17.0.8:6379
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
DB_PORT_6379_TCP_ADDR=172.17.0.8
DB_PORT_6379_TCP_PROTO=tcp
SHLVL=1
HOME=/
container=lxc
_=/usr/bin/env

Debug Iptables

The simplest way is use LOGGING target.

OR use the TRACE target if you want to know more details: http://backreference.org/2010/06/11/iptables-debugging/

Debug ebtables

Use eatables log watcher extension, for example:

$sudo ebtables -t broute -A BROUTING --log-level 6 --log-ip --log-prefix "TRACE: eb:broute:BROUTING" -j ACCEPT
$sudo ebtables -t nat -A OUTPUT --log-level 6 --log-ip --log-prefix "TRACE: eb:nat:OUTPUT"  -j ACCEPT 

Refers

Iptables

http://www.iptables.info/en/iptables-contents.html

http://inai.de/images/nf-packet-flow.png

http://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg (SVG version)

http://linux-ip.net/nf/nfk-traversal.png (Iptables only)

eatables

http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html