Docker blocking network of existing LXC containers

[thesourcerer8]

I am running a server with LXC containers on it.
I found a software that was available as Docker containers, so I installed Docker on the server and tried that software.

• This is reporting unexpected problematic behaviour
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

Both LXC containers should work normally, and new Docker containers should work.
If it isn't possible for Docker to start a container without interrupting existing containers due to any conceptual issue, I would expect Docker to detect that such a problem exists and to provide an overridable error message or at least a warning, that it might interfere with existing containers.

Actual behavior

The Docker containers work, but the LXC containers are suddenly disconnected from the network. The LXC containers are still running, and I can attach to them directly, but they aren't visible on the network anymore, they cannot be pinged, and their services aren't available anymore. Docker has disconnected the running LXC containers from the network.

Steps to reproduce the behavior

Install Debian Linux
Install LXC
Create a LXC container with it's own IP address
Run a server on the LXC container
Install Docker
Download and run a Docker container
Try to access the server in the LXC container over the network

Output of docker version:

Client:
 Version:      17.07.0-ce
 API version:  1.31
 Go version:   go1.8.3
 Git commit:   8784753
 Built:        Tue Aug 29 17:42:54 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.07.0-ce
 API version:  1.31 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   8784753
 Built:        Tue Aug 29 17:41:44 2017
 OS/Arch:      linux/amd64
 Experimental: false

Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 4
Server Version: 17.07.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 42
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3addd840653146c90a254301d6c3a663c7fd6429
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-3-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.41GiB
Name: servername
ID: ZQY3:PKBB:X7ZY:WY54:VKG4:IILY:42LG:KURS:4MXC:JR67:A6IX:T3KT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://proxy:8080/
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)
Linux version 4.9.0-3-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)

[aaronp24]

I ran into this yesterday and the problem seems to be how Docker messes with the FORWARD filter table in iptables:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 431K 1126M DOCKER-USER  all  --  any    any     anywhere             anywhere            
 431K 1126M DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere            
 219K 1090M ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere            
 211K   37M ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere   

These rules block all forwarded packets that aren't to, from, or between Docker containers. My LXC containers use a "br0" interface that's separate from the "docker0" bridge.

[ghost]

@aaronp24, how did you get it working?

[aaronp24]

I just manually flushed the FORWARD table and changed its default policy to ACCEPT

iptables -F FORWARD
iptables -P FORWARD ACCEPT

[aaronp24]

Oh wait, I forgot that I also found you could disable this behavior by putting this in /etc/docker/daemon.json:

{
	"iptables": false
}

[gmcclins]

@aaronp24 Thank you!

I can confirm that /etc/docker/daemon.json:

{
        "iptables": false
}

works. After install of docker and prior to the setting lxc ls shows no ipv4 information. After the setting I now have an ipv4 address again.

Edit: I guess I spoke a little too soon. While this does restore lxc networking, it seems docker is not able to reach the internet now. I've deleted the /etc/docker/daemon.json file I created and reloaded docker. This of course led to lxc container loosing it's address again. I then

iptables -F FORWARD
iptables -P FORWARD ACCEPT

and have confirmed I now have networking in both docker and lxc.

[aaronp24]

I think this fixed it for me:

iptables -t nat -A POSTROUTING -s $dockernet -o ! docker0 -j MASQUERADE

[gmcclins]

@aaronp24

iptables -F FORWARD
iptables -P FORWARD ACCEPT

fixed it for me. Should I be using

iptables -t nat -A POSTROUTING -s $dockernet -o ! docker0 -j MASQUERADE

instead?

[gmcclins]

iptables -t nat -A POSTROUTING -s $dockernet -o ! docker0 -j MASQUERADE

Bad argument `docker0'

[theseer]

Try using SNAT:

iptables -t nat -A POSTROUTING -s $dockernet -o your_uplink_dev -j SNAT --to _your_uplink_ip

As long as you do not try to have publicly reachable services on docker, that should work to have a docker container reach the outside world.

[elnull]

@gmcclins
Wrong arguments order. Should be
iptables -t nat -A POSTROUTING -s $dockernet ! -o docker0 -j MASQUERADE

[gmcclins]

@elnull thank you.

[andreheuer]

@elnull
I´m getting the error iptables v1.6.0: host/network !' not found` because the variable $dockernet is not defined. To what should I set the variable?
BR

[elnull]

@andreheuer
For example, if ip addr show dev docker0 | grep inet shows inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 then $dockernet should be 172.17.0.0/16.

[christophhaase]

I set the environment variable $dockernet as described. But it still doesn't work. The lxc container is not reachable outside the host. I also tried

sudo iptables -t nat -A POSTROUTING -s $dockernet ! -o docker0 -j SNAT --to ip_adress_of_container

because I use static ip adresses for the host and the container. But still no success. The only commands that makes the container reachable are

iptables -F FORWARD
iptables -P FORWARD ACCEPT

So what am I doing wrong?

[Cretection]

Is there a solution?

After every reboot, I have to set

iptables -F FORWARD
iptables -P FORWARD ACCEPT

again

[Psycho0verload]

I've got a permanent solution using
IPv4 Forwarding

[Wilhelmsson177]

I have the same problem as @Cretection the link provided by @Psycho0verload does not help, because IPv4 Forwarding is already enabled. Is there a good overall solution for fixing issues with LXC container and Docker?

[Cretection]

I solved it with edit the /etc/network/interfaces and add
post-up iptables -F FORWARD && iptables -P FORWARD ACCEPT
to the end.
I found this later that my comment as @Psycho0verload not working.

[Cretection]

At last I got one more:
sudo nano /etc/sysctl.conf
uncomment: net.ipv4.ip_forward=1

[mattelacchiato]

All solutions described here did not work in case when restart=always is set for the docker container. I've compared the iptables in case of working and non-working, but they are identical but this line is present when it works:
# Warning: iptables-legacy tables present, use iptables-legacy to see them

Although this output is the same for both cases (working & non-working):

$ sudo iptables-legacy -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I've observed that the order is important: When starting the LXC container first and then the docker containers, it works. Other way around not.

My current workaround is to change /lib/systemd/system/docker.service and add an After dependency to the service which wraps my LXC container.

[kveroneau]

Hmm, by the sounds of this issue not being acknowledged by the docker developers, it seems that using Docker alongside another container technology such as LXC is not really supported. It would be nice if they made a small update to how Docker handles the FORWARD table to allow for new/existing rules to still work as expected. Mind you, I do have Docker running on a Linux router/gateway machine in my home, and I have some services set in the FORWARD table, and those custom iptables entries are unaffected. I wonder if perhaps the issue is related to how LXC is managing it's bridge interface, and uses the FORWARD table in an non-conventional way. I've been wanting to play around with LXC, so I may take a deeper look into what exactly is going on here, and causing LXC to lose it's connection. If I do manage to figure anything out at all, I'll be sure to post it into this issue for others who might try running both container technologies in unison. I do not see any reason why both container technologies could not run together on the same host.

I never ran into any issues on Debian 9 using a NAT behind a lxcbr0 bridge device. Here's what I did, I first stopped the docker daemon, cleared out everything docker placed into my iptables(shouldn't docker clear it's IP tables for me when it's stopped?), set the FORWARD to ACCEPT. Then, I proceeded to install lxc, lxcfs, and dnsmasq-base. bridge-utils was already installed. I configured lxc-net according to the documentation, and started that systemd service. This created my bridge interface, and added some rules into my iptables.

-A FORWARD -o lxcbr0 -j ACCEPT
-A FORWARD -i lxcbr0 -j ACCEPT

I then proceeded to create an lxc container, started it, and confirmed networking both into the container, and networking out to the world wide web was working as expected. Once that was all confirmed, I proceeded to start the docker daemon, where it then re-created it's own iptables rules. I then confirmed that both my docker and lxc containers had networking both inbound and outbound. Both container technologies appear to run on the same host together with no issues. I haven't tried a full host bridge network, where the LXC containers are fully exposed on the host's external network, but I am sure this will work even better as it shouldn't need to add any firewall rules, unless you need your LXC container to communicate back to your host over the network for some reason.

[the-real-ed]

I encountered the same issue, and was able to resolve it after reading the Docker documentation on how to setup Docker on a router.

In short, you need to add your custom rules to the DOCKER-USER table -- it appears that Docker respects these rules (i.e. doesn't clobber them).

For example, for my bridge interface br0, I added the following iptables rule:

-I DOCKER-USER -i br0 -o br0 -j ACCEPT

[mattelacchiato]

Thank you @the-real-ed ! That is really helpful! How do you make this rule persistent?

[the-real-ed]

@mattelacchiato in Debian-based systems, you can install the iptables-persistent package, and then add the rules to /etc/iptables/rules.v4.

The contents of the file should be something along the lines of:

*filter
-I DOCKER-USER -i br0 -o br0 -j ACCEPT
COMMIT

[nameless66]

Update System

sudo apt update && sudo apt upgrade

Install iptables-persistent

sudo apt-get install iptables-persistent # Save current rules for v4 & v6

Add Docker on a router iptables rule

sudo nano /etc/iptables/rules.v4
*filter
:DOCKER-USER - [0:0]
-I DOCKER-USER -i br0 -o br0 -j ACCEPT
COMMIT

Bind lxc.network.link to br0

sudo sed -i '/lxc.network.link/c\lxc.network.link = br0' /etc/piVCCU3/lxc.config

[hoxu]

I ran into some networking issues while using lxc-net and installing docker, as well.

I'm not entirely sure what caused the networking issues, because after looking at the FORWARD chain rules, it should not matter whether lxc-net.service or docker.service is started first, either way ip filter FORWARD chain should end up in a working state.

$ sudo systemctl restart nftables lxc-net docker
$ sudo nft -s list chain ip filter FORWARD
table ip filter {
        chain FORWARD {
                type filter hook forward priority 0; policy drop;
                counter jump DOCKER-USER
                counter jump DOCKER-ISOLATION-STAGE-1
                oifname "docker0" ct state related,established counter accept
                oifname "docker0" counter jump DOCKER
                iifname "docker0" oifname != "docker0" counter accept
                iifname "docker0" oifname "docker0" counter accept
                oifname "lxcbr0" counter accept
                iifname "lxcbr0" counter accept
        }
}

Regardless, it may be a good idea to manually add lxc-net's FORWARD rules to DOCKER-USER chain as well.

Docker and iptables: Add iptables policies before Docker’s rules

If you need to add rules which load before Docker’s rules, add them to the DOCKER-USER chain. These rules are applied before any rules Docker creates automatically.

[mcx]

I solved it with edit the /etc/network/interfaces and add
post-up iptables -F FORWARD && iptables -P FORWARD ACCEPT
to the end.

At last I got one more:
sudo nano /etc/sysctl.conf
uncomment: net.ipv4.ip_forward=1

I can confirm that after performing both steps suggested by @Cretection networking is restored for my LXC containers, and Docker networking remains unaffected after a reboot.

[erik78se]

I got struck by this hard. I've got a thread here that shows the situation and also how I got networking back for lxd.

https://discuss.linuxcontainers.org/t/containers-fails-to-reach-internet/14671

[starkeepers]

The current solution to this problem in the LXD documentation doesn't work on Debian bookworm:

iptables -I DOCKER-USER -i <network_bridge> -o <external_interface> -j ACCEPT

I had to make it bidirectional for networking to start working again, in my case:

iptables -I DOCKER-USER -i lxdbr0 -o eno0 -j ACCEPT
iptables -I DOCKER-USER -i eno0 -o lxdbr0 -j ACCEPT

To make it stick across reboots, it can be added to /etc/network/interfaces as mentioned above, ie:

iface eno0 inet static
  ...
  post-up iptables -I DOCKER-USER -i lxdbr0 -o eno0 -j ACCEPT
  post-up iptables -I DOCKER-USER -i eno0 -o lxdbr0 -j ACCEPT

[simondeziel]

I had to make it bidirectional for networking to start working again, in my case:

FYI, we've updated the LXD documentation to have the returning traffic go through. This is checking the connection state rather than allowing all inbound traffic.

[seanblanchfield]

I struggled a bit with this over the last few days, so here's some info on what finally worked for me. I am running Debian 11 on an Intel NUC with both Docker and LXD installed, and LXD containers configured with bridge networking so that they get regular LAN IP addresses via DHCP, and can communicate normally with the host.

Entirely flushing FORWARD rules out of iptables per @gmcclins advice worked, but seemed a bit heavy-handed. I tried rules like those suggested above by @starkeepers, but they didn't work for me (perhaps due to differences between managed bridges and unmanaged bridges?). To analyse it I captured some log output from iptables showing what traffic in/out of a container normally looks like (using a logging rule like -A FORWARD -j LOG --log-prefix "FORWARDING: ").

This logged tens of thousands of packets in just a few seconds. I grepped out relevant traffic using the container's
host interface as revealed by lxc info <containername> | grep "Host interface" (in this case, "veth68e15422").
Here is some relevant log output for the container's DHCP request and response:

IN=br0 OUT=br0 PHYSIN=veth68e15422 PHYSOUT=eno1 MAC=ff:ff:ff:ff:ff:ff:00:16:3e:a5:d2:ef:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=64 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308 
IN=br0 OUT=br0 PHYSIN=eno1 PHYSOUT=veth68e15422 MAC=00:16:3e:a5:d2:ef:a0:04:60:2f:cd:4d:08:00 SRC=10.0.0.1 DST=10.0.0.6 LEN=335 TOS=0x00 PREC=0x00 TTL=64 ID=63378 PROTO=UDP SPT=67 DPT=68 LEN=315 

Incidentally, some cross-container broadcast and multicast traffic also happens and looks like this:

IN=br0 OUT=br0 PHYSIN=veth68e15422 PHYSOUT=vethf8286751 MAC=ff:ff:ff:ff:ff:ff:00:16:3e:a5:d2:ef:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=64 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308 

I explored adapting the style of rule suggested by @starkeepers by using the physdev module:

iptables -I DOCKER-USER -m physdev --physdev-in eno1 --physdev-out veth68e15422 -j ACCEPT

This would have led to a lot of new rules to manage. I ended up settling on a single broader rule like @the-real-ed suggested above:

iptables -I DOCKER-USER -i br0 -o br0 -j ACCEPT

I also tried to make this rule persist across reboots using a post-up directive in my /etc/network/interfaces file as detailed by @Cretection and others above, but this did not work reliably for me:

auto br0
iface br0 inet dhcp
    bridge-ifaces eno1
    bridge-ports eno1
    up ip link set eno1 up
    post-up iptables -I DOCKER-USER -i br0 -o br0 -j ACCEPT

I had errors in syslog like the following, showing that the DOCKER-USER iptables chain did not yet exist when the interface was coming up.

Apr 11 12:13:08 ifup[1523]: iptables: No chain/target/match by that name.
Apr 11 12:13:08 ifup[1135]: ifup: failed to bring up br0

To resolve this I added a new systemd unit to run after docker starts.

I created a script at /usr/local/bin/post-docker.sh:

#!/usr/bin/env bash

date > /var/log/post-docker-timestamp
iptables -I DOCKER-USER -i br0 -o br0 -j ACCEPT

I created a systemd unit at /etc/systemd/system/postdocker.service:

[Unit]
Description=Post Docker
After=docker.service
BindsTo=docker.service
ReloadPropagatedFrom=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/post-docker.sh
ExecReload=/usr/local/bin/post-docker.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

And enabled it as follows:

sudo systemctl enable postdocker.service

So far, this seems to be working reliably.

[th0ma7]

@seanblanchfield thnx for your solution. What worked on my end what to adapt /usr/local/bin/post-docker.sh such as:

#!/usr/bin/env bash

date > /var/log/post-docker-timestamp
iptables -I DOCKER-USER -i lxdbr0 -j ACCEPT
iptables -I DOCKER-USER -o lxdbr0 -j ACCEPT

And confirmed fixed permanently, it works after a reboot using Ubuntu 23.10.

[jkotran]

This bug blocks DHCP requests from LXC containers and Qemu/KVM VMs on a Debian 12 Bookworm machine. : - /

This allows it through as others have shared:

iptables -I DOCKER-USER -i vmbr0 -j ACCEPT
iptables -I DOCKER-USER -o vmbr0 -j ACCEPT

[Thor-x86]

For anyone who has DOCKER-USER missing in iptables, modify /usr/local/bin/post-docker.sh to:

#!/usr/bin/env bash

date > /var/log/post-docker-timestamp
iptables -N DOCKER-USER
iptables -I FORWARD -j DOCKER-USER
iptables -A DOCKER-USER -j RETURN
iptables -I DOCKER-USER -i br0 -j ACCEPT
iptables -I DOCKER-USER -o br0 -j ACCEPT

Where br0 is your network bridge, change it if necessary.

Source: #810 (comment)

[melbeltagy]

Following the official documentation for LXD https://documentation.ubuntu.com/lxd/en/latest/howto/network_bridge_firewalld/#prevent-connectivity-issues-with-lxd-and-docker fixes the problem.

I applied "Enable IPv4 forwarding" and "Allow egress network traffic flows".

[lengau]

Looks like we missed the 7th birthday of this bug. Happy belated birthday, #103!

[akerouanton]

FWIW, we've some refactoring of our iptables rules cooking in, and it should solve this issue (although that wasn't the main goal). Hopefully, it'll be released in v28.0 (ETA mid-November).