Docker blocking network of existing LXC containers

[thesourcerer8]

I am running a server with LXC containers on it.
I found a software that was available as Docker containers, so I installed Docker on the server and tried that software.

• This is reporting unexpected problematic behaviour
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

Both LXC containers should work normally, and new Docker containers should work.
If it isn't possible for Docker to start a container without interrupting existing containers due to any conceptual issue, I would expect Docker to detect that such a problem exists and to provide an overridable error message or at least a warning, that it might interfere with existing containers.

Actual behavior

The Docker containers work, but the LXC containers are suddenly disconnected from the network. The LXC containers are still running, and I can attach to them directly, but they aren't visible on the network anymore, they cannot be pinged, and their services aren't available anymore. Docker has disconnected the running LXC containers from the network.

Steps to reproduce the behavior

Install Debian Linux
Install LXC
Create a LXC container with it's own IP address
Run a server on the LXC container
Install Docker
Download and run a Docker container
Try to access the server in the LXC container over the network

Output of docker version:

Client:
 Version:      17.07.0-ce
 API version:  1.31
 Go version:   go1.8.3
 Git commit:   8784753
 Built:        Tue Aug 29 17:42:54 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.07.0-ce
 API version:  1.31 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   8784753
 Built:        Tue Aug 29 17:41:44 2017
 OS/Arch:      linux/amd64
 Experimental: false

Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 4
Server Version: 17.07.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 42
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3addd840653146c90a254301d6c3a663c7fd6429
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-3-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.41GiB
Name: servername
ID: ZQY3:PKBB:X7ZY:WY54:VKG4:IILY:42LG:KURS:4MXC:JR67:A6IX:T3KT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://proxy:8080/
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)
Linux version 4.9.0-3-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)

[aaronp24]

I ran into this yesterday and the problem seems to be how Docker messes with the FORWARD filter table in iptables:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 431K 1126M DOCKER-USER  all  --  any    any     anywhere             anywhere            
 431K 1126M DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere            
 219K 1090M ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere            
 211K   37M ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere   

These rules block all forwarded packets that aren't to, from, or between Docker containers. My LXC containers use a "br0" interface that's separate from the "docker0" bridge.

@aaronp24, how did you get it working?

[aaronp24]

I just manually flushed the FORWARD table and changed its default policy to ACCEPT

iptables -F FORWARD
iptables -P FORWARD ACCEPT

[aaronp24]

Oh wait, I forgot that I also found you could disable this behavior by putting this in /etc/docker/daemon.json:

{
	"iptables": false
}

[gmcclins]

@aaronp24 Thank you!

I can confirm that /etc/docker/daemon.json:

{
        "iptables": false
}

works. After install of docker and prior to the setting lxc ls shows no ipv4 information. After the setting I now have an ipv4 address again.

Edit: I guess I spoke a little too soon. While this does restore lxc networking, it seems docker is not able to reach the internet now. I've deleted the /etc/docker/daemon.json file I created and reloaded docker. This of course led to lxc container loosing it's address again. I then

iptables -F FORWARD
iptables -P FORWARD ACCEPT

and have confirmed I now have networking in both docker and lxc.

[aaronp24]

I think this fixed it for me:

iptables -t nat -A POSTROUTING -s $dockernet -o ! docker0 -j MASQUERADE

[gmcclins]

@aaronp24

iptables -F FORWARD
iptables -P FORWARD ACCEPT

fixed it for me. Should I be using

iptables -t nat -A POSTROUTING -s $dockernet -o ! docker0 -j MASQUERADE

instead?

[gmcclins]

iptables -t nat -A POSTROUTING -s $dockernet -o ! docker0 -j MASQUERADE

Bad argument `docker0'

[theseer]

Try using SNAT:

iptables -t nat -A POSTROUTING -s $dockernet -o your_uplink_dev -j SNAT --to _your_uplink_ip

As long as you do not try to have publicly reachable services on docker, that should work to have a docker container reach the outside world.

[elnull]

@gmcclins
Wrong arguments order. Should be
iptables -t nat -A POSTROUTING -s $dockernet ! -o docker0 -j MASQUERADE

[gmcclins]

@elnull thank you.

[andreheuer]

@elnull
I´m getting the error iptables v1.6.0: host/network !' not found` because the variable $dockernet is not defined. To what should I set the variable?
BR

[elnull]

@andreheuer
For example, if ip addr show dev docker0 | grep inet shows inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 then $dockernet should be 172.17.0.0/16.

[christophhaase]

I set the environment variable $dockernet as described. But it still doesn't work. The lxc container is not reachable outside the host. I also tried

sudo iptables -t nat -A POSTROUTING -s $dockernet ! -o docker0 -j SNAT --to ip_adress_of_container

because I use static ip adresses for the host and the container. But still no success. The only commands that makes the container reachable are

iptables -F FORWARD
iptables -P FORWARD ACCEPT

So what am I doing wrong?

[Cretection]

Is there a solution?

After every reboot, I have to set

iptables -F FORWARD
iptables -P FORWARD ACCEPT

again