Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker bypasses ufw firewall rules #690

Open
2 of 3 tasks
binaryfire opened this issue Jun 5, 2019 · 101 comments
Open
2 of 3 tasks

Docker bypasses ufw firewall rules #690

binaryfire opened this issue Jun 5, 2019 · 101 comments

Comments

@binaryfire
Copy link

binaryfire commented Jun 5, 2019

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

Hi all!

ufw in ubuntu should be treated as the "master" when it comes to low level firewall rules (like firewalld in rhel). However docker bypasses ufw completely and does it's own thing with iptables. It was only by chance (luckily!) we discovered this. Example:

ufw deny 8080 (blocks all external access to port 8080)
docker run jboss/keycloak

Expected behaviour: the Keycloak container should be available at port 8080 on localhost/127.0.0.1, but not from the outside world.

Actual behavior

UFW reports port 8080 as blocked but the keycloak docker container is still accessible externally on port 8080.

There is a workaround (https://www.techrepublic.com/article/how-to-fix-the-docker-and-ufw-security-flaw/) however I think techrepublic are correct when then describe it as a "security flaw", and it's a pretty serious one. Most people using ubuntu user ufw. I imagine a large number of them are unaware their UFW rules are being bypassed and all their containers are exposed.

Is this something that can be addressed in the next update? That article was published in Jan 2018.

@binaryfire binaryfire changed the title Docker --net=host bypasses ufw firewall rules Docker bypasses ufw firewall rules Jun 5, 2019
@cpuguy83
Copy link
Collaborator

cpuguy83 commented Jun 5, 2019

The problem is ufw does it's own thing here. The best thing to do here would be to insert a jump rule into the DOCKER-USER chain which will forward to the ufw chain.
There is a pretty lengthy discussion on this in github.com/moby/moby, though (search is failing me, unfortunately).

Note that in your example, docker is not doing anything with iptables OR networking since it's using --net=host.

@binaryfire
Copy link
Author

binaryfire commented Jun 5, 2019

@cpuguy83 Thanks for the quick reply.

With --net=host specified docker (latest version) is still opening the port via iptables, at least on my ubuntu 18.04 fresh install. If that's not supposed to be happening, maybe it's a bug? I agree it definitely shouldn't be doing anything with iptables or networking if --net=host is specified.

I'll see if I can find the thread you mentioned. Perhaps the docker install process could automatically add DOCKER_OPTS="--iptables=false" if ufw is enabled?

@Nutomic
Copy link

Nutomic commented Jul 1, 2019

I just came across the same article myself, and I am very surprised by this behaviour. I dont know all the details about Linux networking, but is there any reason to be doing it? I never heard of any other program that goes around the firewall. If this is some kind of feature, it should definitely disabled by default because it opens up the entire server.

@binaryfire
Copy link
Author

binaryfire commented Jul 2, 2019

@Nutomic I agree. I don't think looking at this as a "UFW problem" is the right approach. The way UFW manages the firewall is quite elegant, which is why the majority of Ubuntu users are using it over firewalld.

I really think the docker devs need to add UFW compatibility ASAP as it's a serious security issue. Or include a clear warning on install letting users know their UFW rules will be ignored and instructions on a workaround.

@lonix1
Copy link

lonix1 commented Sep 6, 2019

@Nutomic ...which is why the majority of Ubuntu users are using it...

+100 All of us are using ufw.

This is an exploit waiting to be exploited.

This should be considered a security risk. Why is it not given priority?

@lonix1
Copy link

lonix1 commented Sep 7, 2019

Does someone know how to report security risks?

In other projects there is usually an email address (or some other mechanism) to report exploits directly to management.

If a server is compromised and docker can be blamed somehow, it would be a major PR headache (at best) for docker management, so I'm sure they'd want to know about this immediately. And if it cannot be fixed, I'm also sure they'd quickly update the docs with major warnings about this problem to let users know about it and legally pass the blame onto us. We should not discover this problem by accident, it should be made clear in the docs - and how best to work around it.

@cpuguy83
Copy link
Collaborator

cpuguy83 commented Sep 7, 2019

There is a SECURITY.md in the moby repo explaining it.
This is not an exploit, however it is easy to misconfigure.

I don't think the desired outcome should be to default to iptables=false but rather have a way to have docker insert its jump rule into ufw.

@binaryfire
Copy link
Author

binaryfire commented Sep 7, 2019

@cpuguy83 IMHO the fact that it's so easy to misconfigure makes it a pretty serious security risk. Other than third party firewall software, I don't know of any other packages that bypass UFW rules. It's such a low-level OS function that most users, even advanced ones, wouldn't think to check.

Definitely agree re: implementation though. Getting Docker to insert its jump rule into UFW would be great.

@lonix1
Copy link

lonix1 commented Sep 7, 2019

Thanks to @cpuguy83, it's:

security @ docker . com

I recommend we all send a message there.

@binaryfire
Copy link
Author

I just had a look at the multiple UFW-related issues for moby and they've all been closed... :/ Does anyone know if podman has the same problem? If not, might be a viable alternative

@lonix1
Copy link

lonix1 commented Sep 7, 2019

That is very surprising! More reason for us to message the security team. This is crazy - how many users don't know they have gaping holes in their security because of this???

For anyone arriving here from google in the future, please do two things:

  • +1 the OP's issue
  • email security @ docker . com

@cpuguy83
Copy link
Collaborator

cpuguy83 commented Sep 7, 2019

btw, I'm think all that's needed is to insert a jump rule from DOCKER-USER to ufw-user-forward

e.g.

sudo iptables -A DOCKER-USER -j ufw-user-forward

@kaysond
Copy link

kaysond commented Sep 9, 2019

Adding some info per @menathor request

See:

Most of these recommend disabling iptables manipulation with --iptables=false and manually configuring the rules as necessary. (i.e. add the following to /etc/ufw/before.rules)

      *nat
      :POSTROUTING ACCEPT [0:0]
      -A POSTROUTING ! -o docker0 -s 172.16.0.0/12 -j MASQUERADE
      COMMIT

More recently, two other workarounds have surfaced which do not use this flag and seem to be more robust:

@lonix1
Copy link

lonix1 commented Sep 9, 2019

Someone from the docker team please tell us the official and secure way to deal with this problem?

I don't want to become an iptables expert just so I can use docker.

@cpuguy83
Copy link
Collaborator

cpuguy83 commented Sep 9, 2019

@lonix1 Did you try sudo iptables -A DOCKER-USER -j ufw-user-forward?

@lonix1
Copy link

lonix1 commented Sep 9, 2019

There are about a dozen approaches starting from 2013/2014, and changing with each major version of docker. At this point I have no understanding which to use, and why. I know ufw well, but not iptables.

I'm hesitant to use the mindless copy-paste approach as I'm afraid to blow up my servers's security. That's why I would be grateful for official guidance, and explanation.

The code you posted seems good, but I have no idea what it does 😃 Would you mind telling us in your opinion which is the best way (I assume it's what you posted above), and why/how it works?

(The simplest approach I've found is not to change anything, but use 127.0.0.1:8080:80 and expose the 8080 via nginx, but even then I'm not sure if that's the best approach, though it seems "cleanest".)

Thanks for helping us out!

@cpuguy83
Copy link
Collaborator

cpuguy83 commented Sep 9, 2019

TMK nothing has changed in a very long time.
I do not expect the default handling of this to change either as it effects many many users.

Docker creates an iptables chain called DOCKER-USER, this is where users can add their own filtering logic, this gets run before port forwarding rules.
The above command puts a jump rule fro, DOCKER-USER to ufw-user-input, which means anything that hits DOCKER-USER (which should be anything Docker related) will get passed along to the ufw-user-forward chain which is where ufw rules should go.

@lonix1
Copy link

lonix1 commented Sep 9, 2019

@cpuguy83 So if I understand correctly, if I implement your approach, thereafter I can continue to use ufw to open/close ports, and never need to mess around with iptables at all? That sounds perfect.

On another note, since you are obviously an expert in this matter, how do you feel this approach compares to the one I posted above (127.0.0.1:8080:80 + nginx) - do you feel one is better/safer/ whatever than the other?

@cpuguy83
Copy link
Collaborator

cpuguy83 commented Sep 9, 2019 via email

@kaysond
Copy link

kaysond commented Sep 10, 2019

I do not expect the default handling of this to change either as it effects many many users.

@cpuguy83 I think its less about a change in behavior and more about a documented best practice to make things work securely.

I say this realizing that you can be explicit about which interface/ip to listen on. But if the expected behavior of UFW is that it blocks all incoming traffic except where specified, then to avoid mistakes, it should do so for Docker as well. Even if it takes some extra configuration.

Also your jump rule sudo iptables -A DOCKER-USER -j ufw-user-forward doesn't work. The first rule in the DOCKER-USER chain is a RETURN (at least in my setup), so it never hits the appended rule. Changing to an insert (sudo iptables -I DOCKER-USER -j ufw-user-forward) fixes that problem but still doesn't seem to work.

@lonix1
Copy link

lonix1 commented Sep 10, 2019

@kaysond What workaround are you using at the moment?

@binaryfire
Copy link
Author

binaryfire commented Sep 10, 2019

EDIT: just fixed my docker / ufw example - the first one didn't make any sense. It's late over here :P

💯 x this:

I think its less about a change in behavior and more about a documented best practice to make things work securely.

Any solution that requires sysadmin knowledge / reconfiguration every time new ports are involved isn't a viable solution. The whole idea behind ufw and docker is ease of use. So taking a step back and breaking it down into super simplistic (i.e. developer) terms:

docker run -p 9000:9000 imnotasysadmin makes imnotasysadmin available on port 9000

I then proxy port 9000 to be behind an oauth gateway on port 443

ufw deny all then ufw allow from 192.168.1.2 to any port 443 means imnotasysadmin should not be accessible on port 9000, and should only be able accessible from 192.168.1.2 via the oauth proxy on 443.

The only reasonable solutions for users like me are:

  1. Docker adds a way of handling this out of the box
  2. Docker provides an official workaroud for ubuntu in their docs (eg. a fixed set of iptables rules) which will handle this in the background and not require modification again after they're set up.

I can't speak for all Ubuntu's users, but I'm a developer, not a sysadmin. I have reasonable linux skills and could implement either of the above solutions no probs. But I no longer feel comfortable using docker on ubuntu in production if my network security is based on hacks from third party sites that may or may not work in certain situations, or may break with future updates.

How can I run things like keycloak, rundeck or other sensitive apps that I'm proxying to 443 and putting behind an oauth proxy if there's any chance docker is going to completely ignore my deny rules and happily expose the original port?

Sorry if this is sounds a bit harsh and peace and love to everyone here, but this has already turned into another "try workaround x" and "workaround x doesn't work when x+y=z" thread. Every one of those on the moby repo have been closed, dating back years.

Developers using Ubuntu (the linux distro with the biggest market share) need an official, supported and documented way of setting docker up so that we can happily docker run and ufw allow / ufw deny all day and everything works as expected.

Or alternatively, official clarification from Docker that "docker does not work with Ubuntu's default firewall". That would be better than silence.

TLDR: https://tenor.com/view/developers-gif-13292051

☮️❤️

@lonix1
Copy link

lonix1 commented Sep 10, 2019

@menathor I'm in the same boat as you exactly. And I'm also wary of the copy-pasta approach.

So I decided to dump ufw and use iptables. Which annoys me, as I have to waste time learning it.

I'm still going through various tutorials. But what surprises me so far, is despite the widely-held belief that iptables is "low-level" and complicated.... it's not. It's simple.

I assume there are edge cases though that you wouldn't typically think of - icmp, specific ports, etc. But if you use a whitelist (rather than blacklist) then at the very worst, you'll make a mistake that'll lock you out of your own server. That's not a problem as you can log in via your hoster's web-based console and fix it, and importantly - the bad guys won't be able to get in either. This is the one and only downside to using iptables, and it's manageable.

Here are some tutorials I'm going through:

More theoretical:

I'll post my final config when I'm done in a few days, if others do the same then we can compare.

@kaysond
Copy link

kaysond commented Sep 10, 2019

@kaysond What workaround are you using at the moment?

@lonix1 Right now, nothing. But I'm working on a larger scale deployment that needs a solution. For now I would suggest reading this: https://github.com/chaifeng/ufw-docker

It has a very descriptive readme, and once you have an idea of what its doing, I'd just run the script. This is a temporary solution until the Docker team comes up with something.

@cpuguy83
Copy link
Collaborator

This is a temporary solution until the Docker team comes up with something

I would say that is the "fix".
At best we could add a flag to check ufw before running docker rules, but it's the same outcome: an iptables rule that jumps chains.
This is the purpose of the DOCKER-USER chain.

@kaysond
Copy link

kaysond commented Sep 10, 2019

I would say that is the "fix".
At best we could add a flag to check ufw before running docker rules, but it's the same outcome: an iptables rule that jumps chains.
This is the purpose of the DOCKER-USER chain.

Well, yes. I think its reasonable to have a set of iptables rules be the "fix," and its easy enough to implement via UFW's config files. But what the community is looking for is something that has been thoughtfully considered by the Docker team and published in the documentation.

For example, the script I linked will allow all traffic from RFC 1918 ranges to reach the Docker network. This is not how UFW behaves by design. If I turn on a service on my Ubuntu host, but don't explicitly allow it in UFW, not even local traffic should reach it.

And maybe that difference is fine. But I, and probably most others, would be more comfortable taking the recommendation from the Docker documentation, and not some guy's github.

@kaysond
Copy link

kaysond commented Sep 16, 2019

Bump. Anyone have any more input? Maybe we need to open an issue on the documentation repo...

@lonix1
Copy link

lonix1 commented Sep 17, 2019

Hey @kaysond Sorry I forgot about this thread. I've been using iptables for a week now and couldn't be happier. For anyone who arrives here, these two links will help you integrate iptables and docker very simply:

If you still want to integrate ufw and docker, then I think what @cpuguy83 wrote above is the way to go (or a variation on that idea). Any rules in the DOCKER-USER chain will be run before docker's own rules, so if there's a jump from there to one of ufw's chains then it MUST work. The question is what order to use, and which one of ufw's many chains to jump to. You'll need to experiment.

Of course the ideal is for the docker team to provide clear guidance of a tested/supported rule, because this isn't something most ufw users know how to do.

@BouDev
Copy link

BouDev commented Feb 4, 2022

This problem has been going on for years. I'm also still waiting for a fix from Docker. I also use this workaround, as I use UFW by default in Ubuntu: https://github.com/chaifeng/ufw-docker

Yeah looked at it. Lot of hassle to keep using an "uncomplicated" firewall :D. In my case I just switched back to Iptables. But still, apart from this nice feature. It at least would be nice to get a "REDALERT!!!" somewhere along the line from UFW or in a log file.

@picode7
Copy link

picode7 commented Feb 13, 2022

Got hacked because I trusted the ufw. Now all data is gone and got published. At least I have backups.

tgbugs added a commit to tgbugs/dockerfiles that referenced this issue Aug 24, 2022
add port forwarding for sparcron server REMINDER/WARNING: docker is
STILL BROKEN for this docker/for-linux#690
@BlueskyFR
Copy link

This is becoming a joke as the years are passing 😅

@danielfree19
Copy link

Why is it still a thing?
Why docker writes to iptables automatically and we dont have a setting if it is a desired option?

@amirshnll
Copy link

btw, I'm think all that's needed is to insert a jump rule from DOCKER-USER to ufw-user-forward

e.g.

sudo iptables -A DOCKER-USER -j ufw-user-forward

work it

@grandeto
Copy link

grandeto commented Jan 22, 2023

Here's is a relatively simple way to make ufw generated iptables rules take precedence over the docker ones

sudo nano /etc/ufw/after.rules
inside *filter section append the following 2 lines just before the # End required lines, like this:

:DOCKER-USER - [0:0]
:ufw-user-input - [0:0]
# End required lines

then insert the following rule -I DOCKER-USER -j ufw-user-input -m comment --comment "forward to ufw" just before COMMIT, like this:

# Forward DOCKER-USER chain to ufw-user-input chain
-I DOCKER-USER -j ufw-user-input -m comment --comment "forward to ufw"

# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT

reboot and validate the DOCKER-USER chain now has the ufw forward rule at position 1:

sudo iptables -t filter -L DOCKER-USER --line-numbers -n -v

the output should look like this:

Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1      104 40743 ufw-user-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* forward to ufw */
2       48 32821 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

test if your use-cases match against the ufw defined rules

EDIT:

Setup DOCKER-USER chain to be rebuild on ufw stop (to prevent ufw start failure after ufw disable/enable)

sudo nano /etc/ufw/before.init

    stop)
        # typically required
        iptables -F DOCKER-USER || true
        iptables -I DOCKER-USER -j RETURN || true
        ;;

sudo chmod +x /etc/ufw/before.init

sudo ufw disable && sudo ufw enable

@zhen-huan-hu
Copy link

test if your use-cases match against the ufw defined rules

Using ufw-user-input could potential cause problems as explained here: https://github.com/chaifeng/ufw-docker

@3dprogramin
Copy link

I setup docker compose for mongodb on CentOS 7 with ufw as firewall, was surprised to see the mongodb accessible from the outside, without adding any ufw rule for it.

The easy (and safe if you ask me) solution I'm using is specifying the IP address on which to bind as well.

27017:27017 to 127.0.0.1:27017:27017

However you put it, it's 2023 and this still has big security implications.
At least on docker installation on an OS with ufw installed and enabled, there should be big warning in red to let the users know about this.

@swfsql
Copy link

swfsql commented Feb 27, 2023

Can one make root-mode docker "simply" behave like root-less docker in regards to the networking rules?
(I had to switch to root-mode, and got surprised by this iptables behavior..)

@Darkzarich
Copy link

Just learned about this issue. I installed Crowdsec on my server and all banned IPs still can request to my services. What a joke 😞

@Darkzarich
Copy link

Darkzarich commented Mar 6, 2023

To anyone facing the same issue while setting up fail2ban or Crowdsec do the following

Crowdsec:

sudo iptables -I FORWARD -m set --match-set crowdsec-blacklists src -j DROP

For fail2ban something similar to this

sudo iptables -I FORWARD 1 -p tcp -j f2b-HTTPS

In fact, fail2ban has its own "targets" in iptables, you can find them all with

sudo iptables -L | grep f2b

It will place an entry in iptables before Docker rule that will reject requests before they reach Docker rule (that allows them all)

@madiele
Copy link

madiele commented Mar 11, 2023

I only now discovered the issue because I found a MINER inside one of my docker containers that exploited ports of a database service I though were under ufw firewall that was meant to be accessed only via personal VPN.

Why is this a thing, I'm dumbfound...

@zhen-huan-hu
Copy link

zhen-huan-hu commented Mar 13, 2023 via email

@UplandsDynamic
Copy link

Yeah, it has been for many years. The lack of official resolution to this security issue is just unbelievable. I guess we have to wait until a high-profile user got hacked because of this.

Being that the Docker maintainers appear so unresponsive to this issue, I'd suggest it might be worth bringing it to the attention of the infosec research community. I recently pinged Steve Gibson, of the well known 'Security Now' podcast. If others do likewise, he may be inclined to take a look and perhaps cover it on the show, to at least shine some publicity on this and alert his audience to the potential risks. His Twitter handle is @SGgrc.

@J4gQBqqR
Copy link

I switched to Arch recently. One of the most important reason for doing so is to configure my firewall from ground up without preconfigured ufw in place. Thank you docker for always having me check, double check and triple check my firewall settings.

And I dare not put my server under DMZ thanks to you. I have no confidence in you messing up my firewalls. I am not capable enough to use iptables and even under firewalld I always felt insecure because how you handle ufw. Could you please gain yourselves some credit by cleaning up this mess?

@burntoc
Copy link

burntoc commented Mar 14, 2023 via email

@MathewJohn1414
Copy link

You can configure Docker to use the ufw firewall by creating a new file /etc/docker/daemon.json if it doesn't exist, and add the following content to it:

{
  "iptables": false,
  "ip6tables": false
}

Then, restart the Docker service using the following command:

sudo systemctl restart docker

This will tell Docker to disable its own iptables rules and use the system's ufw firewall instead.

@g4z
Copy link

g4z commented Apr 1, 2023

can you point to any documentation or other source for this information?

@UplandsDynamic
Copy link

You can configure Docker to use the ufw firewall by creating a new file /etc/docker/daemon.json

@g4z: Here is the official documentation, FWIW:

Prevent Docker from manipulating iptables
It is possible to set the iptables key to false in the Docker engine’s configuration file at /etc/docker/daemon.json, but this option is not appropriate for most users. It is not possible to completely prevent Docker from creating iptables rules, and creating them after-the-fact is extremely involved and beyond the scope of these instructions. Setting iptables to false will more than likely break container networking for the Docker engine.

Source: https://docs.docker.com/network/iptables/#prevent-docker-from-manipulating-iptables

The entire situation is a steaming hot mess.

This repo is legacy, so doubtful if anyone's actually bothering to check here; the README refers users to the Moby upstream repo, to report issues, here:

https://github.com/moby/moby

Security issues should be reported direct to the team, at: security@docker.com

@akerouanton
Copy link
Member

I created a discussion about what's going on with custom iptables rules (whether created through ufw or manually) and what we can do to improve that. It's available here:

@chrisgilbert
Copy link

I just discovered this issue on Rocky Linux 8, using portainer to setup docker compose . I've been using docker for years professionally, but only either on AWS, my mac, and behind other external firewalls. I discovered it when setting up a new dedicated server for Nextcloud and other home stuff.

It seems that the iptables rules break any setup in firewalld, so without manual changes to DOCKER-USER chain you are unable to block docker ports on the internet.

What an atrocious and badly thought through 'feature'. If I'm a professional DevOps Engineer and I didn't realise this after using docker for 10 years or so, I don't know what hope a casual user has.

@matthewblott
Copy link

matthewblott commented Jul 29, 2023

@chrisgilbert as I say above, I'd advise running Docker as rootless. It's fairly straightforward to setup, works with Portainer and gives you peace of mind! Unfortunately some things like Caprover install Docker so it's not always an option but if your setup is pretty vanilla you should be okay.

@arfedulov
Copy link

arfedulov commented Aug 6, 2023

I've just got my database stolen. The database port was supposed to be closed by ufw but it wasn't. Lukilly, the database was not very important.

For those who use docker compose. The following is the solution that seems to work for me. In docker-compose.yml when mapping host ports to container ports specify also ip address part IPADDR:HOSTPORT:CONTAINERPORT. docs . E.g.

"ports":
    - "127.0.0.1:8001:8001"

This will hide host port from the outside world.

@yrro
Copy link

yrro commented Aug 7, 2023

I've just got my database stolen.

Good learning opportunity. 😉

  • Don't run docker as root unless you need it.
  • Don't enable iptables support in the Docker daemon unless you need it.
  • Don't publish a port without binding to 127.0.0.1.

There's a more general point too: if you are running a service and don't want to make it available to other networks, check!

@EssentialRecruit
Copy link

This is 2023 about to crossover to 2024 and am still experiencing this issue
This is a serious security flaw or bug as the case maybe.

Please @docker do something

cooldracula added a commit to planetary-social/ansible-scripts that referenced this issue Jan 30, 2024
@dmuensterer
Copy link

dmuensterer commented Feb 3, 2024

I've just got my database stolen.

Good learning opportunity. 😉

  • Don't run docker as root unless you need it.
  • Don't enable iptables support in the Docker daemon unless you need it.
  • Don't publish a port without binding to 127.0.0.1.

There's a more general point too: if you are running a service and don't want to make it available to other networks, check!

While I 100% agree with your points made, this is still a confession of failure by docker and waiting for disasters to happen.
There is no reasoning as to why docker has to behave like it does. Docker is used by millions and them not being able to patch such a simple design flaw in almost 5 years is just sad.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests