docker and ufw serious problems

[phlegx]

Having installed ufw and blocking all incoming traffic by default (sudo ufw default deny) by running docker images that map the ports to my host machine, these mapped docker ports are accessible from outside, even though they are never allowed to be accessed.

Please note that on this machine DEFAULT_FORWARD_POLICY="ACCEPT" as described on this page http://docs.docker.io/en/latest/installation/ubuntulinux/#ufw has not been enabled and the property DEFAULT_FORWARD_POLICY="DROP" is still set.

Any ideas what might causing this?

Output of ufw status:

$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing)
New profiles: skip

To                         Action      From
--                         ------      ----
22                         ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
5666                       ALLOW IN    95.xx.xx.xx
4949                       ALLOW IN    95.xx.xx.xx
22                         ALLOW IN    Anywhere (v6)
443/tcp                    ALLOW IN    Anywhere (v6)
80/tcp                     ALLOW IN    Anywhere (v6)

Here is the output of my rabbitmq via docker ps:

cf4028680530        188.xxx.xx.xx:5000/rabbitmq:latest           /bin/sh -c /usr/bin/   5 weeks ago         Up 5 days           0.0.0.0:15672->15672/tcp, 0.0.0.0:5672->5672/tcp   ecstatic_darwin/rabbitmq,focused_torvalds/rabbitmq,rabbitmq,sharp_bohr/rabbitmq,trusting_pike/rabbitm

Nmap test:

nmap -P0 example.com -p 15672

Starting Nmap 5.21 ( http://nmap.org ) at 2014-03-18 11:27 CET
Nmap scan report for example.com (188.xxx.xxx.xxx)
Host is up (0.048s latency).
PORT      STATE SERVICE
15672/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds

General infos:

• Ubuntu 12.04 server

$ uname -a
Linux production 3.8.0-29-generic #42~precise1-Ubuntu SMP Wed Aug 14 16:19:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client version: 0.9.0
Go version (client): go1.2.1
Git commit (client): 2b3fdf2
Server version: 0.9.0
Git commit (server): 2b3fdf2
Go version (server): go1.2.1
Last stable version: 0.9.0

$ docker info
Containers: 12
Images: 315
Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 339
WARNING: No swap limit support

[Soulou]

Ufw is only setting things in the filter table. Basically, the docker traffic is diverted before and goes through the nat table, so ufw in this case is basically useless, if you want to drop the traffic for a container you need to add rules in the mangle/nat table.

http://cesarti.files.wordpress.com/2012/02/iptables.gif

[honi]

@Soulou would you recommend adding to mangle or nat?

edited previous comment after some research

[honi]

In my case I wanted to only allow a specific IP to connect to the exposed port. I've managed to do this with this rule.

It drops all connections to port <Port> if source IP is not <RemoteIP>. I suppose that if you would want to completely block all connections, then simply remove the ! -s <RemoteIP> bit.

iptables -I PREROUTING 1 -t mangle ! -s <RemoteIP> -p tcp --dport <Port> -j DROP

[cpuguy83]

@honi In Docker 1.5 (maybe 1.4?) there were several iptables changes. Can you verify if this is still a problem with 1.5?

[saidimu]

@cpuguy83 I can confirm that this is still a problem with Docker 1.6

Adding --iptables=false to DOCKER_OPTS enables the expected behavior.

[lauralorenz]

+1, still a problem with Docker 1.6

[cpuguy83]

ping @mavenugo @mrjana

[newhook]

So what is the story here? With Docker version 1.7.0, build 0baf609 on Ubuntu 14 this is still completely broken. Also the installaton instructions on https://docs.docker.com/installation/ubuntulinux/ have a section "Enable UFW forwarding" which appears to be unnecessary. Anyone installing docker on an Ubuntu box exposes any forwarded ports from their containers to the outside world, and even worse looking at the ufw rules gives no hints that this is occurring which is needless to stay pretty bad.

[VascoVisser]

Also with Docker 1.7 here. My experience is that Docker+UFW can facilitate two scenarios.

The first scenario and default behavior indeed exposes all mapped ports to the outside world; UFW cannot filter access to the containers.

Alternatively when setting the --iptables=false option, filtering incoming traffic with UFW works as expected. However, doing this stops the containers from making outbound connections to the outside world. Inter container communication still works. If you don't need outbound connectivity, then UFW together with --iptables=false seems to be a viable solution.

In my opinion a sensible default behavior for docker would be how it behaves currently with --iptables=false and allow outbound connections from the containers (or possibly make this easily configurable via a config option).

[newhook]

I don't have a problem getting out. Did you try:

ufw allow in on docker0

[VascoVisser]

@newhook ufw allow in on docker0 doesn't work for me. Even with ufw disabled I can't get out with --iptables=false.

[VascoVisser]

I have been experimenting with this a few hours now. I think I got it figured out.

... the installaton instructions on https://docs.docker.com/installation/ubuntulinux/ have a section "Enable UFW forwarding" which appears to be unnecessary.

The FORWARD chain does need policy set to ACCEPT if you have --iptables=false. It only appears this is not needed because the Docker installation package auto starts Docker and adds iptable rules the FORWARD chain. When afterwards you add --iptables=false to your config and restart docker those rules are still there. After the next reboot these rules will be gone and your containers wont be able to communicate unless you have the FORWARD chain policy set to ACCEPT.

What you need for a setup that allows filtering with UFW, inter container networking and outbound connectivity is

• start docker with --iptables=false
• FORWARD chain policy set to ACCEPT
• add the following NAT rule:
  iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE

[newhook]

You are indeed correct! After a reboot communication is gone. Those rules seem to sort everything out. Thanks very much!

[dakky]

start docker with --iptables=false
FORWARD chain policy set to ACCEPT
add the following NAT rule:
iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE

with this setup its not possible anymore to access exposed ports from within a container:

• Container 1: exposed port 12345
• login to Container 2:
  telnet 172.17.42.1 12345 does not work anymore

[VascoVisser]

@dakky I can't reproduce your issue. I have no issues with inter container communication. I suggest making sure you expose the port in your Dockerfile. Also try flushing your iptables rules and delete all user-defined chains before configuring and enabling UFW.

In any case it would be good if someone from the Docker team can verify that the configuration I propose makes sense.

[include]

Hi, any update on this? I can't find any official source how to fix this.
Currently I have a simple setup like:

/etc/defaults/ufw: DEFAULT_FORWARD_POLICY="ACCEPT"
/etc/defaults/docker: DOCKER_OPTS="--iptables=false"

ufw enable
ufw allow 22/tcp
ufw deny 80/tcp
ufw reload

host# docker run -it --rm -p 80:8000 ubuntu bash
container# apt-get update
container# python3 -m http.server

.1 I can reach Internet from container
.2 Internet can reach container via public-address:80

Am I missing something here? 10x

[teodor-pripoae]

I had managed to fix this through iptables mangle. The first 2 lines are optional if you want to allow access to some ports on eth1 (private network, if it exists).

sudo iptables -t mangle -A FORWARD -i eth1 -o docker0 -j ACCEPT
sudo iptables -t mangle -A FORWARD -i docker0 -o eth1 -j ACCEPT
sudo iptables -t mangle -A FORWARD -i docker0 -o eth0 -j ACCEPT
sudo iptables -t mangle -A FORWARD -i eth0 -o docker0 -j ACCEPT -m state --state ESTABLISHED,RELATED
sudo iptables -t mangle -A FORWARD -i eth0 -o docker0 -j DROP

[lenovouser]

This is still a problem. Is there a clear fix available? I don't expect a built-in solution. But maybe some iptables or or nat rules? I don't feel like testing all possible solutions in this issue now just to brick my system 😄

[mikehaertl]

Guys, this is a serious security issue. Why is there no hint in the documentation for it? Only by accident I found out, that my MySQL Port is wide open to the world. I absolutely didn't expect that as I've used ufw before and it was reliable enough to not spend another thought on it. So I trusted the advice to change the forward policy to ACCEPT. I would never have expected that it basically completely suspends ufw.

[mikehaertl]

For the record, the solution from @VascoVisser worked for me with docker V1.10. Here are the files I had to change:

• Set DEFAULT_FORWARD_POLICY="ACCEPT" in /etc/default/ufw

• Set DOCKER_OPTS="--iptables=false" in /etc/default/docker

• Add the following block with my custom bridge's ip range to the top of /etc/ufw/before.rules:

  # nat Table rules
  *nat
  :POSTROUTING ACCEPT [0:0]
  
  # Forward traffic from eth1 through eth0.
  -A POSTROUTING -s 192.168.0.0/24 -o eth0 -j MASQUERADE
  
  # don't delete the 'COMMIT' line or these nat table rules won't be processed
  COMMIT
  

Note: I'm using a custom network for my docker containers, so you may have to change the 192.168.0.0 above to match your network range. The default is 172.17.0.0/16 as in Vasco's comment above.

UPDATE: On Ubuntu 16.04 things are different, because docker is started by systemd, so /etc/default/docker is ignored. The solution described here creates the file /etc/systemd/system/docker.service.d/noiptables.conf with this content

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --iptables=false

and issue systemctl daemon-reload afterwards.

[lenovouser]

@mikehaertl I want to mention that this is not really an issue just with UFW, as it is just another layer over iptables. This is a general problem in my opinion.

[mikehaertl]

@lenovouser Thing is, that the documentation has some recommendation which sounds like "do this and everything is fine with ufw". But that's definitely not the case, so there should be big warning signs there.

[chaifeng]

Let me re-explain it in a simple way ^_^

If there is a Linux server:

• Install an HAProxy server on the server, and listen on port 80
• Run the command ufw allow 80 to allow the public access the HAProxy.

Create an httpd container, mapping the host's port 8080 to the httpd container's port 80.

docker run --rm -d --name httpd -p 8080:80 httpd:alpine

If using ufw-user-input chain, the httpd container will be exposed by default. Because the httpd container's port is same as the HAProxy server's, it's 80.

if using ufw-user-forward chain, the httpd container is still private. We can use the command ufw route allow 80 to expose the httpd container later.

[mikehaertl]

@chaifeng Thanks for your detailled explanations. One question to your last example:

So if you now connect from outside to your host on port 8080 you'll reach your httpd container, even though you never issued ufw allow 8080? Is that correct? In that case I see your point.

[chaifeng]

Correct, ufw allow 8080 or ufw deny 8080 has no effect on accessing the httpd container, if we use ufw-user-input chain.

docker run --rm -d --name httpd -p 8080:80 httpd:alpine

[rubot]

To prevent ufw startup problems, we use the before.init. All in all targeting DOCKER_USER chain into ufw-user-input was our solution also, and works well enough. Despite we are not using any other nat rules, just leave that table alone.

docker_ufw_setup=https://gist.githubusercontent.com/rubot/418ecbcef49425339528233b24654a7d/raw/docker_ufw_setup.sh
bash <(curl -SsL $docker_ufw_setup)
# Reset and open port 22
RESET=1 bash <(curl -SsL $docker_ufw_setup)
DEBUG=1 bash <(curl -SsL $docker_ufw_setup)

https://gist.github.com/rubot/418ecbcef49425339528233b24654a7d

[mikehaertl]

@rubot @tsuna As @chaifeng showed your solution is not bullet proof. I try to sum it up in my own words:

• You have a host service public to the world with ufw allow 123 (123 is an arbitrary port)
• You have a container that by default also listens on 123
• You map port 123 from that container to port 456 on the host
• Now your host port 456 is also open to the public even though you never added a rule for that in ufw

[rubot]

you map port 123 from that container to port 456 on the host

this should end up in an nat rule setup by docker, as docker is using masquerading.
all docker nat rules end up in DOCKER_USER input chain, which will drop all not explictly allowed ports.

-N DOCKER-USER
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-USER -m conntrack --ctstate INVALID -j DROP
-A DOCKER-USER -i eth0 -j ufw-user-input
-A DOCKER-USER -i eth0 -j DROP

[rubot]

can´t confirm that:

root@dev ~ # iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -o docker_gwbridge -m addrtype --src-type LOCAL -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER-INGRESS -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.2:80
-A DOCKER-INGRESS -j RETURN

root@dev ~ # iptables -S DOCKER-USER
-N DOCKER-USER
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-USER -m conntrack --ctstate INVALID -j DROP
-A DOCKER-USER -i eth0 -j ufw-user-input
-A DOCKER-USER -i eth0 -j DROP
-A DOCKER-USER -j RETURN

root@dev ~ # ufw status
Status: active

To                         Action      From
--                         ------      ----
80,443/tcp                 ALLOW       Anywhere

root@dev ~ # docker run --rm -p 8000:80 jwilder/whoami
Listening on :8000

→ curl dev:8000
curl: (7) Failed to connect to dev port 8000: Connection refused

[rubot]

ah, had an error

root@dev ~ # docker run --rm -p 8000:80 nginx

curl dev:8000
<!DOCTYPE html>
...

thanks, will check that

[chaifeng]

Hi @rubot

At the beginning of this thread, @Soulou has a comment #4737 (comment)

Ufw is only setting things in the filter table. Basically, the docker traffic is diverted before and goes through the nat table, so ufw in this case is basically useless, if you want to drop the traffic for a container you need to add rules in the mangle/nat table.

http://cesarti.files.wordpress.com/2012/02/iptables.gif

[chaifeng]

Hi @rubot

I found your typo, the port of jwilder/whoami is 8000, not 80.

docker run --rm -p 8000:80 jwilder/whoami

should be

docker run --rm -p 9999:8000 jwilder/whoami

curl dev:9999

Thanks!

[rubot]

As we don´t have a rule in ufw-user-input that is allowing port 8000, this prevents as expected.
The problems are only, as you stated, with allowed ports, in ufw-user-input, which correspond to the containerside exposed port when publishing to an arbitrary hostside port. The problem didn´t show up for me as I only have a reduced set of 1:1 mappings, using docker swarm.
Thanks again for that hint.

[chaifeng]

One more thing I noticed, you modified the file /etc/ufw/before.init to create a chain. You don't need to do this.

As UFW use iptables-restore to restore rules from the file /etc/ufw/after.rules. Any new chain defined in this file will be created.

For example, add the following lines to the end of after.rules. The new chain ufw-docker will be created after restarting UFW.

*filter
:ufw-docker - [0:0]
:DOCKER-USER - [0:0]

-A DOCKER-USER -j ufw-docker

COMMIT

[rubot]

This crashed a lot of times, as MANAGE_BUILTINS=no is set for ufw. I decided to be more aggressive in cleaning up the rules manually.

[rubot]

At the beginning of this thread, @Soulou has a comment #4737 (comment)

Quickfix: https://gist.github.com/rubot/418ecbcef49425339528233b24654a7d#file-docker_ufw_setup-sh-L152

[rubot]

At the beginning of this thread, @Soulou has a comment #4737 (comment)

Quickfix: https://gist.github.com/rubot/418ecbcef49425339528233b24654a7d#file-docker_ufw_setup-sh-L152

This fix works in the first place as expected, so far.
Unfortunately it has the downside, the origin ip is lost, that was provided by host mode, using nginx stream and proxy_protocol

I tried to workaround with a static ip for the stream instance, but static ips for ingress are not supported, yet:

• https://docs.docker.com/compose/compose-file#ipv4_address-ipv6_address
• #31860 (comment)

[rubot]

Because I have to finish it and can´t switch to something different, or even fiddle with custom docker iptables, a cronjob will help for retrieving the origin ip again

https://gist.github.com/rubot/418ecbcef49425339528233b24654a7d#file-docker_ufw_setup-sh-L55

Edit:
Talking about nat table -
Changed cron to only allow 1:1 port mappings for DOCKER chain.
DOCKER-INGRESS chain seems to be save, as there are only 1:1 port mappings
This affected container and services running under mode=host, which both get DNAT rules created in the DOCKER chain

[xhafan]

@rubot @tsuna As @chaifeng showed your solution is not bullet proof. I try to sum it up in my own words:

* You have a host service public to the world with `ufw allow 123` (123 is an arbitrary port)

* You have a container that by default also listens on `123`

* You map port `123` from that container to port `456` on the host

* Now your host port `456` is also open to the public even though you never added a rule for that in 

@mikehaertl, I could not replicate it, your scenario works fine for me, i.e. host port 456 is not publicly opened.

[mikehaertl]

@xhafan I did not try it either and just tried to sumarize what @chaifeng found out. From a cursory look at the chains it sounded reasonable to me. Maybe @chaifeng can comment?

[chaifeng]

@rubot @tsuna As @chaifeng showed your solution is not bullet proof. I try to sum it up in my own words:

* You have a host service public to the world with `ufw allow 123` (123 is an arbitrary port)

* You have a container that by default also listens on `123`

* You map port `123` from that container to port `456` on the host

* Now your host port `456` is also open to the public even though you never added a rule for that in 

@mikehaertl, I could not replicate it, your scenario works fine for me, i.e. host port 456 is not publicly opened.

@xhafan

Make sure that ufw doesn't allow port 28080 first, but allows port 80.

Run the command to start an httpd container and publish the container port 80 on the host port 28080.

docker run -d --rm --name httpd -p 28080:80 httpd:alpine

We can access port 28080 via the IP address of the host from outside.

Even ufw deny 28080 cannot block accessing this httpd container from outside.

[xhafan]

We can access port 28080 via the IP address of the host from outside.

Even ufw deny 28080 cannot block accessing this httpd container from outside.

@chaifeng, I tried what you suggested, and I can confirm that it opened the port 28080 from the outside. It does that also for nginx container. But, for some reason, jekyll container, which publishes a port too, is not opened from the outside. Here is my docker-compose.yml:

version: '3.5'
services:

  jekyll:
	image: jekyll/jekyll:3.8.3
	container_name: jekyll
	command: jekyll serve --force_polling
	ports:
	  - 28081:4000

  httpd:
	image: httpd:alpine
	container_name: httpd
	ports:
	  - 28080:80

28080 is open from the outside, but 28081 is not. That's why it gave me an impression that it's a working solution. Any idea why jekyll's published port is not opened from the outside?

[mikehaertl]

Make sure that ufw doesn't allow port 28080 first, but allows port 80.

@xhafan Did you see this? You probably have 80 open, that's why 28080 is also open for your httpd container. In your jekyll case port 4000 must be open on the host. Then 28081 would get opened implicitely, too.

[chaifeng]

We can access port 28080 via the IP address of the host from outside.
Even ufw deny 28080 cannot block accessing this httpd container from outside.

@chaifeng, I tried what you suggested, and I can confirm that it opened the port 28080 from the outside. It does that also for nginx container. But, for some reason, jekyll container, which publishes a port too, is not opened from the outside. Here is my docker-compose.yml:

version: '3.5'
services:

  jekyll:
	image: jekyll/jekyll:3.8.3
	container_name: jekyll
	command: jekyll serve --force_polling
	ports:
	  - 28081:4000

  httpd:
	image: httpd:alpine
	container_name: httpd
	ports:
	  - 28080:80

28080 is open from the outside, but 28081 is not. That's why it gave me an impression that it's a working solution. Any idea why jekyll's published port is not opened from the outside?

I think the port 4000 is not allowed in UFW on your host. That the port 28080 is open is because the container port of httpd is 80, and port 80 is allowed on the host.

Allow port 4000, and you will find the port 28081 is open.

sudo ufw allow 4000

[xhafan]

@mikehaertl, @chaifeng thanks for the explanation. That it quite weird behaviour, however the whole solution works for me. One needs to be aware of it though.

[jojoob]

@tsuna I like your solution. But if I apply it the first time I disable and enable the ufw to apply the changes everything works as expected. But if I then reload the ufw or execute ufw disable/enable again I get the following error from ufw and ufw is then inactive.:

$ sudo ufw reload
ERROR: Could not load logging rules
$ sudo ufw status verbose
Status: inactive

The problem goes away if I comment out the rule -A DOCKER-USER -i eth0 -j ufw-user-input. But of course this rule is required to make user defined rules work.
If I set MANAGE_BUILTINS=yes in /etc/default/ufw it is also possible to restart/reload ufw. But after ufw has restarted I must also restart docker service to fix docker's iptable rules.
Disabling logging in /etc/ufw/ufw.conf with LOGLEVEL=off has no effect.

Edit: I know think I understand what's happening. The default setting for MANAGE_BUILTINS is no. Which means that ufw will not touch any chains except its own. But by adjusting the after.rules like @tsuna suggests we are changing other chains. Now ufw can't clean up the rules correctly.
I have decided to set MANAGE_BUILTINS to yes as a solution.

[vald-phoenix]

I don't know maybe it's not relevant already but when I create /etc/docker/daemon.json with such content:

{"iptables": false}

And restart docker sudo systemctl restart docker it starts to work without any additional efforts, so ports no longer available to the world.

So the question do I miss something or this is just fine?