Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Best Practices for Docker and UFW #777

Open
kaysond opened this issue Sep 5, 2019 · 25 comments
Open

Best Practices for Docker and UFW #777

kaysond opened this issue Sep 5, 2019 · 25 comments

Comments

@kaysond
Copy link

kaysond commented Sep 5, 2019

There is a well known security concern when running Docker on a Ubuntu host that uses UFW as its main firewall: Docker's manipulation of iptables bypasses the rules created by UFW, enabling access by default to containers with ports mapped despite UFW being enabled. Though there are existing methods of securing the network (e.g. binding to 127.0.0.1), the extra security of UFW can be desirable.

See:

Most of these recommend disabling iptables manipulation with --iptables=false and manually configuring the rules as necessary.

More recently, two other workarounds have surfaced which do not use this flag and seem to be more robust:

These are at least a year old now, and despite the many results it's still not common knowledge as searches continue: https://trends.google.com/trends/explore?q=docker%20ufw&geo=US

It seems that the Docker team isn't interested in addressing this on their end, so the purpose of this issue is to request community feedback, determine best practices, and create a PR to hopefully add something to the documentation.

@binaryfire
Copy link

binaryfire commented Sep 7, 2019

Hi @kaysond, there's an existing thread about this here: #690. If you could add this info there it would be great

@AkihiroSuda
Copy link

AkihiroSuda commented Sep 9, 2019

Rootless mode seems also useful as a workaround for this

@binaryfire
Copy link

binaryfire commented Sep 10, 2019

Hey @AkihiroSuda, thanks a lot for your suggestion. The only posts I've found referencing rootless mode are using nightly builds so it seems this is an experimental feature at the moment and not ready for production? Are there any official docs and stable builds available?

Also would you be able to expand on the pros and cons of rootless vs regular modes or direct me to some articles which explain this clearly?

Thanks for your time. Appreciate any insights you might have to offer :)

@AkihiroSuda
Copy link

AkihiroSuda commented Sep 11, 2019

The only posts I've found referencing rootless mode are using nightly builds

Not only for nightly builds
https://download.docker.com/linux/static/stable/x86_64/docker-rootless-extras-19.03.2.tgz

it seems this is an experimental feature at the moment and not ready for production

Yes, it is likely to remain experimental until we can complete migration to cgroup2 (around the end of the year)

Also would you be able to expand on the pros and cons of rootless vs regular modes or direct me to some articles which explain this clearly?

https://www.slideshare.net/AkihiroSuda/dockercon-2019-hardening-docker-daemon-with-rootless-mode

@shinebayar-g
Copy link

shinebayar-g commented Oct 4, 2020

Hi guys, I just made a little script that solves the ufw firewall and docker iptables issue. You can take a look at repo here. It's heavily based on chaifeng/ufw-docker's ufw rules, but automates the manual process. All you have to do is run the container with special label UFW_MANAGED=TRUE. Please let me know your feedbacks. Of course code is crap, but hopefully it works.

@pySilver
Copy link

pySilver commented Jan 25, 2021

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

@kaysond
Copy link
Author

kaysond commented Jan 26, 2021

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

I actually like this a lot more than https://github.com/chaifeng/ufw-docker
Seems like it properly interfaces ufw to docker iptables rules without unncessary assumptions.

@nathan-march
Copy link

nathan-march commented Feb 3, 2021

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

Something to be aware of this solution (and every other one I've found so far) is that they're filtering the dest-port after the nat translation has been applied. This means if you have two containers listening on 443 internally but exposed on different external ports, they'll both be open.

In my case I have a pub container on 443->443 and an admin container on 443->8443, so none of these workarounds manage to close off that 8443 port.

@shinebayar-g
Copy link

shinebayar-g commented Feb 3, 2021

@nathan-sav I'm using ufw-docker. You can give container a static ip address and write specific rule for that IP. In that way your problem can be solved. For example, I have multiple mysql containers running on default 3306, exposed on 3306, 3307, 3308 externally, and they all have different ufw rules. Unfortunately containers' ip address must be static, otherwise IP specific rules doesn't make sense. To solve that problem, I started ufw-docker-automated.

@nathan-march
Copy link

nathan-march commented Feb 3, 2021

@nathan-sav I'm using ufw-docker. You can give container a static ip address and write specific rule for that IP. In that way your problem can be solved. For example, I have multiple mysql containers running on default 3306, exposed on 3306, 3307, 3308 externally, and they all have different ufw rules. Unfortunately containers' ip address must be static, otherwise IP specific rules doesn't make sense. To solve that problem, I started ufw-docker-automated.

Yeah unfortunately that doesn't work for me, I don't want to manage container ips like that and I don't like the idea of needing a script to watch for IP changes and update rules appropriately.

I think I have a better solution in the works, I'll follow up if/once I'm successful :)

@nathan-march
Copy link

nathan-march commented Feb 3, 2021

Ok, here's what I came up with and this works much better for me:

  • You can restart ufw/docker at will and they don't clobber each other
  • Allowed ports only line up to the external ports in docker, allowing 443 in your firewall won't allow access to any port that's mapped to 443 inside a container
  • You do not get to use the ufw tool to manage the allowed ports (I use ansible to build my after.rules with all my rules in it, don't care about the CLI)

It works by tagging the allowed packets in the NAT table and then allowing them later on in the filter table, rather than doing the matching at filter time (when you're dealing with already natted packets).

It all fits into /etc/ufw/after.rules:

*nat
:PREROUTING ACCEPT [0:0]

# Flush rules before we start so things don't get duplicated on ufw reload
-F PREROUTING

###### Here you add the allowed ports for your docker containers ######
-A PREROUTING -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x5/0xffffffff
-A PREROUTING -p tcp -m tcp --dport 443 -j MARK --set-xmark 0x5/0xffffffff

# Debug
#-A PREROUTING -j LOG --log-prefix "NAT-PRE:" --log-level 6

# Replace the docker rule that was here before
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER

# Commit nat table
COMMIT

*filter
:ufw-after-input - [0:0]
:ufw-after-output - [0:0]
:ufw-after-forward - [0:0]
# Need to create the DOCKER-USER chain now, in case docker isn't running yet so the below commands don't fail
:DOCKER-USER - [0:0]

# don't log noisy services by default (UFW DEFAULT)
-A ufw-after-input -p udp --dport 137 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp --dport 138 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp --dport 139 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp --dport 445 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp --dport 67 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp --dport 68 -j ufw-skip-to-policy-input

# don't log noisy broadcast (UFW DEFAULT)
-A ufw-after-input -m addrtype --dst-type BROADCAST -j ufw-skip-to-policy-input

###### Here you add the allowed ports for regular traffic (not docker) ######
-A ufw-after-input -s 216.XXX.XXX.XXX -j ACCEPT
-A ufw-after-input -p TCP --dport 22 -s 10.0.0.0/8 -j ACCEPT

# Flush rules before we start so things don't get duplicated on ufw reload
-F DOCKER-USER

# Just allow outgoing
-A DOCKER-USER -s 172.18.0.0/16 -j ACCEPT

# Debug
#-A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j LOG --log-prefix "ALLOW CONNTRACK:" --log-level 6

# Allow packets after connection gets brought up, since they don't hit the docker nat rules and won't get tagged / allowed
-A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Debug
#-A DOCKER-USER -m mark --mark 0x5 -j LOG --log-prefix "ALLOW DOCKER-USER:" --log-level 6
#-A DOCKER-USER -m mark ! --mark 0x5 -j LOG --log-prefix "DROP DOCKER-USER:" --log-level 6

# Drop anything that didn't get marked or previously accepted
-A DOCKER-USER -m mark ! --mark 0x5 -j DROP

# Replace the default docker rule
-A DOCKER-USER -j RETURN

# Commit filter table
COMMIT

There might be bugs but so far it seems to work reliably, I've restarted docker/ufw a bunch of times with various combinations of them being enabled at boot. Feedback welcome!

@kaysond
Copy link
Author

kaysond commented Feb 3, 2021

Something to be aware of this solution (and every other one I've found so far) is that they're filtering the dest-port after the nat translation has been applied. This means if you have two containers listening on 443 internally but exposed on different external ports, they'll both be open.

This is more a fundamental part of iptables than anything to do with docker. It's just because the filter tables come after nat.

There might be bugs but so far it seems to work reliably, I've restarted docker/ufw a bunch of times with various combinations of them being enabled at boot. Feedback welcome!

Marking is definitely a good workaround (annoying that you can't just drop in nat but such is life). Be careful about -A DOCKER-USER -s 172.18.0.0/16 -j ACCEPT though, because docker can use a variety of subnets, including 192.168.0.0/16

@nathan-march
Copy link

nathan-march commented Feb 4, 2021

This is more a fundamental part of iptables than anything to do with docker. It's just because the filter tables come after nat.

Agreed, I just mentioned it so other people wouldn't be caught off guard after reading this thead.

Marking is definitely a good workaround (annoying that you can't just drop in nat but such is life). Be careful about -A DOCKER-USER -s 172.18.0.0/16 -j ACCEPT though, because docker can use a variety of subnets, including 192.168.0.0/16

Excellent point, I'll see if I can come up with a cleaner way to handle that :)

@kaysond
Copy link
Author

kaysond commented Feb 4, 2021

Excellent point, I'll see if I can come up with a cleaner way to handle that :)

Might just be able to do --src-type LOCAL. Not sure if it handles the veth tunnels though

@pySilver
Copy link

pySilver commented Feb 4, 2021

In my case I have a pub container on 443->443 and an admin container on 443->8443, so none of these workarounds manage to close off that 8443 port.

I've heard about this problem. So in other words when you do
docker run -p 443:443 some_image & docker run -p 443:8443 maybe_different_image both 443 and 8443 of host machine become exposed despite the fact that UFW allows only 443? Is that related to fact that docker image exposes 443 originally or because first image is mapped to 443 ?

@nathan-march
Copy link

nathan-march commented Feb 4, 2021

I've heard about this problem. So in other words when you do
docker run -p 443:443 some_image & docker run -p 443:8443 maybe_different_image both 443 and 8443 of host machine become exposed despite the fact that UFW allows only 443? Is that related to fact that docker image exposes 443 originally or because first image is mapped to 443 ?

Correct.

Iptables filtering rules are applied AFTER the nat rules are applied, so iptables first of all maps 8443 -> 443, then the destination port 443 allow rule lets it through. This is why ufw-docker suggests setting manual ips to your containers and putting those ips into the rules.

@kaysond
Copy link
Author

kaysond commented Feb 4, 2021

@nathan-sav It looks like the conntrack extension might be able to do this in a cleaner way: http://ipset.netfilter.org/iptables-extensions.man.html#lbAO

Check out --ctstate DNAT and --ctorigdstport <port>

@algo7
Copy link

algo7 commented Mar 16, 2021

There's another work around which uses reverse proxy from Nginx or Apache (installed locally) to map ports to containers bind to localhost. However, it adds extra complexities and reduces the flexibilities of using Docker.

@Pimmelton
Copy link

Pimmelton commented Mar 17, 2021

There's another work around which uses reverse proxy from Nginx or Apache (installed locally) to map ports to containers bind to localhost. However, it adds extra complexities and reduces the flexibilities of using Docker.

That's essentially what I ended up doing after quickly tiring of fiddling around with UFW after rules and such, with one simple difference.

I set up a Nginx reverse proxy directly on the host system with a minimal configuration that does nothing but forward all queries (including request headers) directly to a Traefik container bound to a localhost port. Traefik then works as expected, dynamically forwarding to containers as I bring them up and down as if it were being accessed directly. The difference is that my firewall behaves as expected with straightforward ufw commands as in days of yore.

It's working well so far, though I still have to do some fine tuning, like whether to make Nginx the TLS termination proxy or pass that on through to Traefik. I'm inclined to do the former because it suits my particular needs and allows me the flexibility to serve stuff via Nginx directly from the host on another port/virtualhost or whatever.

@gnat
Copy link

gnat commented Mar 18, 2021

As rootless becomes more common, this may solve itself to an extent, but still, gaping security hole for newcomers- this really needs better, sane defaults!

@algo7
Copy link

algo7 commented Mar 20, 2021

As rootless becomes more common, this may solve itself to an extent, but still, gaping security hole for newcomers- this really needs better, sane defaults!

Exactly. It should have mentioned it somewhere on the documentation. I guess ufw is just one of those firewall solutions out there (though quite popular), and Docker thinks that it is not worth mentioning.

@gnat
Copy link

gnat commented Mar 20, 2021

Reminder to people following: ufw is just an iptables wrapper. The issue is Docker installs its own chain at the very start of iptables, which effectively ignores all of your firewall rules.

Putting Dockers chain at the end of iptables, or at least, after ufw's chain could solve this (since ufw has no ability to modify chains other than its own).

@arkroan
Copy link

arkroan commented Apr 22, 2021

Docker already provides integration with Firewalld, respecting the way Firewalld works. I do not think it would be such a stretch to also support UFW and how UFW works, respecting the hierarchy in iptables.

@Na-Cly
Copy link

Na-Cly commented Oct 7, 2021

This really hasn't been addressed?

@algo7
Copy link

algo7 commented Oct 7, 2021

This really hasn't been addressed?

No. At least I see nothing related to this in the last release logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests