Docker container cannot connect to docker host on it's own exposed port if INPUT chain policy is DROP

[tallandtree]

Description

I've got a docker container running in it's own bridge network on a docker host . The containers port 8080 is exposed. I've got iptables default policy on the host set to DROP. Docker daemon manages the docker iptables rules.

If I run curl outside my container to http://:8080 I get HTTP status 200 and the correct website my container is running.

However the same curl from inside my container on that host (to both the IP or hostname) fails to connect:

curl: (7) Failed to connect to on port 8080: Host is unreachable
This doesn't happen when trying this from a Docker container from another host or from the host itself.

I can solve this by adding the following iptables rule:
sudo iptables -I INPUT 1 -i <docker-bridge-interface> -j ACCEPT
Where <docker-bridge-interface> is the name of the bridge interface, in which the docker container is running.

Now I can do curl -v dockerhost:port from my container running in the docker-bridge-interface network and exposing above port on the dockerhost.

I think this should be solved by docker: whenever a bridge network is created, a firewall rule should be set to allow the containers in this network to reach the host on it's own exposed (host) port. Reason we need this is that some webapplications use a baseurl instead relative urls internally.

Steps to reproduce the issue:

$ sudo iptables -P INPUT DROP
$ docker network create -d bridge mynetwork
$ docker run -d -p 8000:8000 --net=mynetwork --name=revealjs amouat/revealjs:latest 
$ docker exec -ti revealjs /bin/bash
# curl -v <dockerhost-ip>:8000
curl: (7) Failed to connect to <dockerhost-ip> port 8000: Connection refused
Or depending on your overall firewall rules, you could also get a timeout.

Describe the results you received:

curl -v http://curie1.ccveu.local:8080
* Rebuilt URL to: http://curie1.ccveu.local:8080/
*   Trying 192.168.56.5...
* connect to 192.168.56.5 port 8080 failed: Connection refused
* Failed to connect to curie1.ccveu.local port 8080: Connection refused
* Closing connection 0
curl: (7) Failed to connect to curie1.ccveu.local port 8080: Connection refused

This doesn't happen when trying this from a Docker container from another host or from the host itself.

Describe the results you expected:

curl -v http://curie1.ccveu.local:8080
* Rebuilt URL to: http://curie1.ccveu.local:8080/
*   Trying 192.168.56.5...
* Connected to curie1.ccveu.local (192.168.56.5) port 8080 (#0)
> GET / HTTP/1.1
> Host: curie1.ccveu.local:8080
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Server: Apache-Coyote/1.1
< X-AREQUESTID: 477x3x1
< Location: /startup.jsp?returnTo=%2Fdefault.jsp
< Content-Type: text/html;charset=UTF-8
< Content-Length: 0
< Date: Thu, 27 Oct 2016 07:57:49 GMT
< 
* Connection #0 to host curie1.ccveu.local left intact

Additional information you deem important (e.g. issue happens only occasionally):

I can solve this by adding an iptables rule to allow connections from docker network to host:
$ sudo iptables -I INPUT 1 <interfacename-mynetwork> -j ACCEPT

However, IMHO docker should add this rule when a bridge or other network is created.

Output of docker version:

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 8
 Running: 6
 Paused: 0
 Stopped: 2
Images: 15
Server Version: 1.12.1
Storage Driver: aufs
 Root Dir: /data/docker/docker/aufs
 Backing Filesystem: extfs
 Dirs: 187
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null bridge host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-28-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.954 GiB
Name: curie.ccveu.local
ID: XKC4:VJXM:XIAR:E7LT:4ELP:G7LS:4KPE:MFIB:T7E4:2YGT:AQCR:IIRE
Docker Root Dir: /data/docker/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://proxy:8080
Https Proxy: http://proxy:8080
No Proxy: .ccveu.local
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 debops.ccv.architecture=Ubuntu 16.04
 debops.ccv.environment=production
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
This is reproducable on the following environments:

• VirtualBox Ubuntu 16.04
• ESX Virtual Machine Ubuntu 16.04
• Virtual Box with Debian Jessie.
• Physical Fedora 19 laptop with docker 1.9.1

[bbayani]

#dibs

[mathroc]

how can I find the value of <interfacename-mynetwork> in sudo iptables -I INPUT 1 <interfacename-mynetwork> -j ACCEPT for a given container ?

[tallandtree]

$ ip route get <Subnet> | awk '{print $4}'

Where is the subnet of the docker network.
Example:

$ ip route get 172.17.0.0 | awk '{print $4}'
docker0

[HcgRandon]

+1 This really needs fixed

[jcon321]

+1

[huima]

+1 Lost some time with this one

[jvrahav]

Could someone explain why the communication between docker and host port fails here. Does http play a role here.

[tallandtree]

nope, it's due to the firewall configuration (iptables) on the docker host.

[jvrahav]

In the same machine i have a total of 4 containers.There is a ha load balancer container and a emqtt container. The traffic from load balancer is able to reach the emqtt container without adding the firewall rule mentioned here as workaround. Any idea why it isnt working between two other containers where one container is making a http call to a service running in another container.

[tallandtree]

Not sure, depends on your exact configuration. But maybe in your case the containers run in the same docker network and the load balancer finds the other containers directly through the container name. It is only a problem if they go via the docker host (http calls usually go via the docker host - dns name resolves to docker host ip address). Probably if they'd use the container name, it would be working as then the traffic remains inside the docker network. I have an apache container running in the same network as my applications and apache also reaches the other containers via the container name directly. The application containers, however, use a baseurl (which resolves to the ip address of the host) and this causes the problem. I cannot set the baseurl to the container name.

[KamilKopaczyk]

I got exact same problem.
2 different docker networks (N1, N2), 2 different externally exposed containers/applications (C1,C2) that need to talk to each other.
HTTP request from machine hosting those containers to C1 can be successfully made, however request from C1 to C2 is timing out. The same request, from other machine, but also from container is also successful.

iptables are managed by docker daemon.

What's the recommended (and working) way of doing this?

[tallandtree]

I wrote some ansible scripts and library (python) to handle this for all servers that need it.

[tdiesler]

It seems that this is still not working ...

$ docker --version
Docker version 1.13.1, build 1ab62f1-unsupported

$ uname -a
Linux MyHost 4.16.6-302.fc28.x86_64 #1 SMP Wed May 2 00:07:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Here a simple test

$ docker run -d -p 95.179.138.139:80:80 httpd:2.4
$ curl 95.179.138.139:80
<html><body><h1>It works!</h1></body></html>

$ docker run -i -t fedora:28 bash
# curl 95.179.138.139:80
curl: (7) Failed to connect to 95.179.138.139 port 80: No route to host

As suggested above, I can fix this by adding

sudo iptables -I INPUT 1 -i docker0 -j ACCEPT

[cobolbaby]

The above method does not work for me. The general solution is that pay attention to userload-proxy configuration.

[bodhi-one]

bump

[polarathene]

Probably can close?: (EDIT: This is a default policy with UFW, at least on Ubuntu 23.04 Vultr VPS)

• Open issue at docker/docs to share advice, or feature request implementing opt-in support instead of bug report?
• Other users encountered similar cross-network connection issues due to the daemon setting userland-proxy: false.
• Upcoming IPVS feature may also be a fix when ready.

---

INPUT with a default policy of DROP is not a common default to encounter for a distro, unless you've got UFW running right? (iptables -P INPUT DROP, caution if in an SSH session and no rule exists to allow it, as you'll lose the connection 😅 )

If you're explicitly configuring this, it's probably better covered by improving documentation. Unless someone wants to propose a way to opt-in such modification and then provides a PR that gets through review.

It could be equally unexpected that Docker made such a modification on it's own (which has been an issue in the past with port publishing bypassing existing firewall management via UFW / firewalld).

---

If you don't have the INPUT DROP policy then as mentioned by an earlier comment, a common issue preventing containers communicating across networks (indirectly via the published ports) is due to having your Docker daemon configured with userland-proxy: false.

You can set userland-proxy: true, or try these temporary iptables fixes and share if they resolve the issue.