docker and ufw serious problems

[phlegx]

Having installed ufw and blocking all incoming traffic by default (sudo ufw default deny) by running docker images that map the ports to my host machine, these mapped docker ports are accessible from outside, even though they are never allowed to be accessed.

Please note that on this machine DEFAULT_FORWARD_POLICY="ACCEPT" as described on this page http://docs.docker.io/en/latest/installation/ubuntulinux/#ufw has not been enabled and the property DEFAULT_FORWARD_POLICY="DROP" is still set.

Any ideas what might causing this?

Output of ufw status:

$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing)
New profiles: skip

To                         Action      From
--                         ------      ----
22                         ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
5666                       ALLOW IN    95.xx.xx.xx
4949                       ALLOW IN    95.xx.xx.xx
22                         ALLOW IN    Anywhere (v6)
443/tcp                    ALLOW IN    Anywhere (v6)
80/tcp                     ALLOW IN    Anywhere (v6)

Here is the output of my rabbitmq via docker ps:

cf4028680530        188.xxx.xx.xx:5000/rabbitmq:latest           /bin/sh -c /usr/bin/   5 weeks ago         Up 5 days           0.0.0.0:15672->15672/tcp, 0.0.0.0:5672->5672/tcp   ecstatic_darwin/rabbitmq,focused_torvalds/rabbitmq,rabbitmq,sharp_bohr/rabbitmq,trusting_pike/rabbitm

Nmap test:

nmap -P0 example.com -p 15672

Starting Nmap 5.21 ( http://nmap.org ) at 2014-03-18 11:27 CET
Nmap scan report for example.com (188.xxx.xxx.xxx)
Host is up (0.048s latency).
PORT      STATE SERVICE
15672/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds

General infos:

• Ubuntu 12.04 server

$ uname -a
Linux production 3.8.0-29-generic #42~precise1-Ubuntu SMP Wed Aug 14 16:19:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client version: 0.9.0
Go version (client): go1.2.1
Git commit (client): 2b3fdf2
Server version: 0.9.0
Git commit (server): 2b3fdf2
Go version (server): go1.2.1
Last stable version: 0.9.0

$ docker info
Containers: 12
Images: 315
Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 339
WARNING: No swap limit support

[Soulou]

Ufw is only setting things in the filter table. Basically, the docker traffic is diverted before and goes through the nat table, so ufw in this case is basically useless, if you want to drop the traffic for a container you need to add rules in the mangle/nat table.

http://cesarti.files.wordpress.com/2012/02/iptables.gif

[honi]

@Soulou would you recommend adding to mangle or nat?

edited previous comment after some research

[honi]

In my case I wanted to only allow a specific IP to connect to the exposed port. I've managed to do this with this rule.

It drops all connections to port <Port> if source IP is not <RemoteIP>. I suppose that if you would want to completely block all connections, then simply remove the ! -s <RemoteIP> bit.

iptables -I PREROUTING 1 -t mangle ! -s <RemoteIP> -p tcp --dport <Port> -j DROP

[cpuguy83]

@honi In Docker 1.5 (maybe 1.4?) there were several iptables changes. Can you verify if this is still a problem with 1.5?

[saidimu]

@cpuguy83 I can confirm that this is still a problem with Docker 1.6

Adding --iptables=false to DOCKER_OPTS enables the expected behavior.

[lauralorenz]

+1, still a problem with Docker 1.6

[cpuguy83]

ping @mavenugo @mrjana

[newhook]

So what is the story here? With Docker version 1.7.0, build 0baf609 on Ubuntu 14 this is still completely broken. Also the installaton instructions on https://docs.docker.com/installation/ubuntulinux/ have a section "Enable UFW forwarding" which appears to be unnecessary. Anyone installing docker on an Ubuntu box exposes any forwarded ports from their containers to the outside world, and even worse looking at the ufw rules gives no hints that this is occurring which is needless to stay pretty bad.

[VascoVisser]

Also with Docker 1.7 here. My experience is that Docker+UFW can facilitate two scenarios.

The first scenario and default behavior indeed exposes all mapped ports to the outside world; UFW cannot filter access to the containers.

Alternatively when setting the --iptables=false option, filtering incoming traffic with UFW works as expected. However, doing this stops the containers from making outbound connections to the outside world. Inter container communication still works. If you don't need outbound connectivity, then UFW together with --iptables=false seems to be a viable solution.

In my opinion a sensible default behavior for docker would be how it behaves currently with --iptables=false and allow outbound connections from the containers (or possibly make this easily configurable via a config option).

[newhook]

I don't have a problem getting out. Did you try:

ufw allow in on docker0

[VascoVisser]

@newhook ufw allow in on docker0 doesn't work for me. Even with ufw disabled I can't get out with --iptables=false.

[VascoVisser]

I have been experimenting with this a few hours now. I think I got it figured out.

... the installaton instructions on https://docs.docker.com/installation/ubuntulinux/ have a section "Enable UFW forwarding" which appears to be unnecessary.

The FORWARD chain does need policy set to ACCEPT if you have --iptables=false. It only appears this is not needed because the Docker installation package auto starts Docker and adds iptable rules the FORWARD chain. When afterwards you add --iptables=false to your config and restart docker those rules are still there. After the next reboot these rules will be gone and your containers wont be able to communicate unless you have the FORWARD chain policy set to ACCEPT.

What you need for a setup that allows filtering with UFW, inter container networking and outbound connectivity is

• start docker with --iptables=false
• FORWARD chain policy set to ACCEPT
• add the following NAT rule:
  iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE

[newhook]

You are indeed correct! After a reboot communication is gone. Those rules seem to sort everything out. Thanks very much!

[dakky]

start docker with --iptables=false
FORWARD chain policy set to ACCEPT
add the following NAT rule:
iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE

with this setup its not possible anymore to access exposed ports from within a container:

• Container 1: exposed port 12345
• login to Container 2:
  telnet 172.17.42.1 12345 does not work anymore

[VascoVisser]

@dakky I can't reproduce your issue. I have no issues with inter container communication. I suggest making sure you expose the port in your Dockerfile. Also try flushing your iptables rules and delete all user-defined chains before configuring and enabling UFW.

In any case it would be good if someone from the Docker team can verify that the configuration I propose makes sense.

[include]

Hi, any update on this? I can't find any official source how to fix this.
Currently I have a simple setup like:

/etc/defaults/ufw: DEFAULT_FORWARD_POLICY="ACCEPT"
/etc/defaults/docker: DOCKER_OPTS="--iptables=false"

ufw enable
ufw allow 22/tcp
ufw deny 80/tcp
ufw reload

host# docker run -it --rm -p 80:8000 ubuntu bash
container# apt-get update
container# python3 -m http.server

.1 I can reach Internet from container
.2 Internet can reach container via public-address:80

Am I missing something here? 10x

[teodor-pripoae]

I had managed to fix this through iptables mangle. The first 2 lines are optional if you want to allow access to some ports on eth1 (private network, if it exists).

sudo iptables -t mangle -A FORWARD -i eth1 -o docker0 -j ACCEPT
sudo iptables -t mangle -A FORWARD -i docker0 -o eth1 -j ACCEPT
sudo iptables -t mangle -A FORWARD -i docker0 -o eth0 -j ACCEPT
sudo iptables -t mangle -A FORWARD -i eth0 -o docker0 -j ACCEPT -m state --state ESTABLISHED,RELATED
sudo iptables -t mangle -A FORWARD -i eth0 -o docker0 -j DROP

[lenovouser]

This is still a problem. Is there a clear fix available? I don't expect a built-in solution. But maybe some iptables or or nat rules? I don't feel like testing all possible solutions in this issue now just to brick my system 😄

[mikehaertl]

Guys, this is a serious security issue. Why is there no hint in the documentation for it? Only by accident I found out, that my MySQL Port is wide open to the world. I absolutely didn't expect that as I've used ufw before and it was reliable enough to not spend another thought on it. So I trusted the advice to change the forward policy to ACCEPT. I would never have expected that it basically completely suspends ufw.

[mikehaertl]

For the record, the solution from @VascoVisser worked for me with docker V1.10. Here are the files I had to change:

• Set DEFAULT_FORWARD_POLICY="ACCEPT" in /etc/default/ufw

• Set DOCKER_OPTS="--iptables=false" in /etc/default/docker

• Add the following block with my custom bridge's ip range to the top of /etc/ufw/before.rules:

  # nat Table rules
  *nat
  :POSTROUTING ACCEPT [0:0]
  
  # Forward traffic from eth1 through eth0.
  -A POSTROUTING -s 192.168.0.0/24 -o eth0 -j MASQUERADE
  
  # don't delete the 'COMMIT' line or these nat table rules won't be processed
  COMMIT
  

Note: I'm using a custom network for my docker containers, so you may have to change the 192.168.0.0 above to match your network range. The default is 172.17.0.0/16 as in Vasco's comment above.

UPDATE: On Ubuntu 16.04 things are different, because docker is started by systemd, so /etc/default/docker is ignored. The solution described here creates the file /etc/systemd/system/docker.service.d/noiptables.conf with this content

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --iptables=false

and issue systemctl daemon-reload afterwards.

[lenovouser]

@mikehaertl I want to mention that this is not really an issue just with UFW, as it is just another layer over iptables. This is a general problem in my opinion.

[mikehaertl]

@lenovouser Thing is, that the documentation has some recommendation which sounds like "do this and everything is fine with ufw". But that's definitely not the case, so there should be big warning signs there.

[pySilver]

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

[LucaGabi]

Put Docker behind UFW

*filter
:DOCKER-USER - [0:0]
:ufw-user-input - [0:0]

-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-USER -m conntrack --ctstate INVALID -j DROP
-A DOCKER-USER -i eth0 -j ufw-user-input
-A DOCKER-USER -i eth0 -j DROP
COMMIT

I get: ERROR: Could not load logging rules after sudo ufw reload

[BenjamenMeyer]

most recent discussion in #22054

[iamdevnull]

We can access port 28080 via the IP address of the host from outside.
Even ufw deny 28080 cannot block accessing this httpd container from outside.

@chaifeng, I tried what you suggested, and I can confirm that it opened the port 28080 from the outside. It does that also for nginx container. But, for some reason, jekyll container, which publishes a port too, is not opened from the outside. Here is my docker-compose.yml:

version: '3.5'
services:

  jekyll:
	image: jekyll/jekyll:3.8.3
	container_name: jekyll
	command: jekyll serve --force_polling
	ports:
	  - 28081:4000

  httpd:
	image: httpd:alpine
	container_name: httpd
	ports:
	  - 28080:80

28080 is open from the outside, but 28081 is not. That's why it gave me an impression that it's a working solution. Any idea why jekyll's published port is not opened from the outside?

I think the port 4000 is not allowed in UFW on your host. That the port 28080 is open is because the container port of httpd is 80, and port 80 is allowed on the host.

Allow port 4000, and you will find the port 28081 is open.

sudo ufw allow 4000

I did exactly this test on my VPS with a PUBLIC-IP (202.xx.xxx.xxx), i cannot reproduce the behaviour why?

To                         Action      From
--                         ------      ----
4000                       ALLOW       Anywhere                  
80                         ALLOW       Anywhere              

 Name               Command               State                 Ports               
------------------------------------------------------------------------------------
httpd    httpd-foreground                 Up      0.0.0.0:28080->80/tcp             
jekyll   /usr/jekyll/bin/entrypoint ...   Up      35729/tcp, 0.0.0.0:28081->4000/tcp

Chain DOCKER (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !br-65d351f468a4 br-65d351f468a4  0.0.0.0/0            172.19.0.3           tcp dpt:4000

Chain ufw-user-input (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4000
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4000

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       tcp  --  !br-65d351f468a4 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:28081 to:172.19.0.3:4000

### Test from another public server with public ip
──╼ #nc -vz -w 1 202.xx.xxx.xxx 28081
nc: connect to 202.xx.xxx.xxx port 28081 (tcp) timed out: Operation now in progress

Any Idea?

[iamdevnull]

We can access port 28080 via the IP address of the host from outside.
Even ufw deny 28080 cannot block accessing this httpd container from outside.

@chaifeng, I tried what you suggested, and I can confirm that it opened the port 28080 from the outside. It does that also for nginx container. But, for some reason, jekyll container, which publishes a port too, is not opened from the outside. Here is my docker-compose.yml:

version: '3.5'
services:

  jekyll:
	image: jekyll/jekyll:3.8.3
	container_name: jekyll
	command: jekyll serve --force_polling
	ports:
	  - 28081:4000

  httpd:
	image: httpd:alpine
	container_name: httpd
	ports:
	  - 28080:80

28080 is open from the outside, but 28081 is not. That's why it gave me an impression that it's a working solution. Any idea why jekyll's published port is not opened from the outside?

I think the port 4000 is not allowed in UFW on your host. That the port 28080 is open is because the container port of httpd is 80, and port 80 is allowed on the host.
Allow port 4000, and you will find the port 28081 is open.

sudo ufw allow 4000

I did exactly this test on my VPS with a PUBLIC-IP (202.xx.xxx.xxx), i cannot reproduce the behavour, why?

To                         Action      From
--                         ------      ----
4000                       ALLOW       Anywhere                  
80                         ALLOW       Anywhere              

 Name               Command               State                 Ports               
------------------------------------------------------------------------------------
httpd    httpd-foreground                 Up      0.0.0.0:28080->80/tcp             
jekyll   /usr/jekyll/bin/entrypoint ...   Up      35729/tcp, 0.0.0.0:28081->4000/tcp

Chain DOCKER (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !br-65d351f468a4 br-65d351f468a4  0.0.0.0/0            172.19.0.3           tcp dpt:4000

Chain ufw-user-input (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4000
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4000

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       tcp  --  !br-65d351f468a4 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:28081 to:172.19.0.3:4000

### Test from another public server with public ip
──╼ #nc -vz -w 1 202.xx.xxx.xxx 28081
nc: connect to 202.xx.xxx.xxx port 28081 (tcp) timed out: Operation now in progress

Any Idea?

Based on chaifeng/ufw-docker#53 after removing these lines:

-A ufw-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "
-A ufw-docker-logging-deny -j DROP

I can confirm the behaviour that the published port 28081 is open after allowing port 4000.

[asifhj]

I did not do any changes to default ufw or docker.
Just I used
ufw route allow ....
instead of ufw allow

And added rule
iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE

[onelittlehope]

With the default install of UFW and Docker, I was able to ensure that external IPs are not able to connect to the published ports of Docker containers via the following changes.

I amended the /etc/ufw/after.rules file and at the end of the file, AFTER the COMMIT line, I added the following:

# Sources:
# - https://serverfault.com/a/933803
# - https://github.com/moby/moby/issues/22054
# - https://github.com/moby/moby/issues/4737
*filter
:DOCKER-USER - [0:0]

# drop invalid packets
-A DOCKER-USER -m conntrack --ctstate INVALID -j DROP

# allow established connections
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

#-------------------------------------------------------------------------------
# a) To allow local containers to communicate with a service on the host:
#
#      Use "ufw allow in ..."   e.g.   "ufw allow in 1234/tcp"
#
# b) To allow allow an external IP (x.x.x.x) to communicate with a container
#    on its published host port (1234):
#
#      -A DOCKER-USER -i eth0 -s x.x.x.x -p tcp -m conntrack --ctorigdstport 1234 --ctdir ORIGINAL -j ACCEPT
#-------------------------------------------------------------------------------

# My test example rule (you don't need this):
-A DOCKER-USER -i eth0 -s 192.168.1.0/24 -p tcp -m conntrack --ctorigdstport 1234 --ctdir ORIGINAL -j ACCEPT

# drop all other incoming packets
-A DOCKER-USER -i eth0 -j DROP

COMMIT

After saving the file, run ufw reload and check the output of iptables-save | grep DOCKER-USER. It should show:

:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A DOCKER-USER -m conntrack --ctstate INVALID -j DROP
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-USER -s 192.168.1.0/24 -i eth0 -p tcp -m conntrack --ctorigdstport 1234 --ctdir ORIGINAL -j ACCEPT
-A DOCKER-USER -i eth0 -j DROP
-A DOCKER-USER -j RETURN

What is important to note is:

• The following line is what allows external IPs in the 192.168.1.0/24 range to access the service I am publishing on the host via a Docker container. (Note: This rule matches the Host port not the container port)

-A DOCKER-USER -s 192.168.1.0/24 -i eth0 -p tcp -m conntrack --ctorigdstport 1234 --ctdir ORIGINAL -j ACCEPT

• To allow local containers on the host to access ports listening on the host, whether they are published Docker container ports or processes running natively on the host, I am issuing commands such as ufw allow in 1234/tcp , ufw allow in 80/tcp.

• The containers are prevented from accessing services on the hosts unless allowed via ufw and that external IPs are prevented from accessing published ports of Docker containers without a rule being added to the DOCKER-USER chain in the /etc/ufw/after.rules file. Control hence lies with the sysadmin at all times.

Does anybody see any issues with the above setup?

[cpuguy83]

Just wanted to update with some thoughts on changes we discussed amongst maintainers:

#45076 sends a warning to clients on container create when port forwards are requested for anything other than localhost with an option to disable via env var (on the daemon).

Also planning to abstract port forwarding logic and provide an implementation that uses IPVS NAT instead of iptables. This would mean dockerd is not poking holes in the firewall. It would be opt-in to not break existing users.

After this interface is well tested we plan to make this available as a plugin (exactly how is not determined yet, maybe an API over a unix socket or just a simple call to a host binary).
There could potentially be an iptables/nftables version of this implementation that takes a better approach than the existing one does as well.

[akerouanton]

I created a discussion about what's going on with custom iptables rules (whether created through ufw or manually) and what we can do to improve that. It's available here:

• Summary of the issues about custom iptables rules #45524

[xamir82]

It's unbelievably irresponsible of Docker to have not fixed this major security gotcha for 9 years. Strange...

[aradalvand]

Also planning to abstract port forwarding logic and provide an implementation that uses IPVS NAT instead of iptables. This would mean dockerd is not poking holes in the firewall. It would be opt-in to not break existing users.

@cpuguy83 I don't think anyone is relying on such an extremely unexpected behavior, and even if they are relying on it unknowingly, they shouldn't be. I would argue this is one of those cases where it's definitely okay to break the current behavior, when the current behavior can be considered a blatant bug by any reasonable definition: why would a program (e.g. Docker) punch holes in the firewall?!

It should not be opt-in IMO, just fix it as a default. This is the sort of thing that plagues many people without them even being aware of it, therefore it's precisely the kind of thing you want to have 'fixed by default', so to speak.

[cpuguy83]

@aradalvand I would expect most people to be relying on this.
I definitely empathize with people who aren't expecting this to break their existing firewall rules, however this is what the feature was designed to do and has been this way for over 10 years now.
Changing the meaning of it without providing options (and replacing it with a relatively untested option) is not feasible.

Can we make the current behavior opt-in at some point? Probably, but we need the to have alternative implementations available and well tested before doing so.

[mstgelais]

In my case I wanted to only allow a specific IP to connect to the exposed port. I've managed to do this with this rule.

It drops all connections to port <Port> if source IP is not <RemoteIP>. I suppose that if you would want to completely block all connections, then simply remove the ! -s <RemoteIP> bit.

iptables -I PREROUTING 1 -t mangle ! -s <RemoteIP> -p tcp --dport <Port> -j DROP

Thanks a lot! In my case, this is exactly what I needed.

[torypages]

Came here because I got kdevtmpfsi malware as a result of this.

edit: To elaborate a bit. If you configure ufw to only allow ssh, the expectation is that only ssh will be allowed regardless of any docker configuration.

Yes, my postgres password should have not been postgres, but this was a toy app and the fact that I got malware actually did not matter. I just remade the container. But it is exposed to me the fact that blocking everything but ssh did in fact not block everything but ssh. In a production scenario maybe I wouldn't have got hit because I'd have a better password, but I would have been blissfully unaware of the fact that my postgres server was being exposed to the internet when I thought it wasn't, and that's bad!

Maybe this is in fact a ufw issue, I don't know. I'm just here because this where the conversation led me to when researching the problem. This is some pretty dangerous stuff.

[SteMoeller]

Yeah... me too... all my databases were stolen und a message from some russian guy wanting btc was left... Thank god that there were no data in the tables...

[lewismunene020]

In my case I wanted to only allow a specific IP to connect to the exposed port. I've managed to do this with this rule.

It drops all connections to port <Port> if source IP is not <RemoteIP>. I suppose that if you would want to completely block all connections, then simply remove the ! -s <RemoteIP> bit.

iptables -I PREROUTING 1 -t mangle ! -s <RemoteIP> -p tcp --dport <Port> -j DROP

GREAT WORK TOU REALLY HELPED ME OUT TODAY

[mitsukuri]

Are there any plans to actually make docker play nice with firewall rules and host's localhost and spare us developers some already huge cognitive load using crutches instead of real solutions and trying to memorize all the madhouse of incompatible permutations of rootless vs rooful, 0.0.0.0 vs 127.0.0.1 vs 10.0.2.2 vs docker.host.internal, -p vs --net=host, DOCKERD_ROOTLESS_ROOTLESSKIT_DISABLE_HOST_LOOPBACK=true|false, slirp4nets, nsenter -U -preserve-credentials bullshit? This has got ridiculous.