Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containers can talk to the host over the network even with ICC disabled #21990

Open
vincentwoo opened this issue Apr 13, 2016 · 28 comments
Open

Containers can talk to the host over the network even with ICC disabled #21990

vincentwoo opened this issue Apr 13, 2016 · 28 comments

Comments

@vincentwoo
Copy link
Contributor

@vincentwoo vincentwoo commented Apr 13, 2016

Output of docker version:

ubuntu@execute-1460530807:~$ docker version
Client:
 Version:      1.11.0-rc5
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   6178547
 Built:        Mon Apr 11 21:07:24 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0-rc5
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   6178547
 Built:        Mon Apr 11 21:07:24 2016
 OS/Arch:      linux/amd64

Output of docker info:

ubuntu@execute-1460530807:~$ docker info
Containers: 17
 Running: 17
 Paused: 0
 Stopped: 0
Images: 23
Server Version: 1.11.0-rc5
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: none
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 3.19.0-58-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 29.45 GiB
Name: execute-1460530807
ID: PQ6T:3IL3:XDMV:KKFC:TAJN:MDAO:6T6K:SSO4:L3YQ:LIDK:NNPZ:6J5U
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
GCE

Steps to reproduce the issue:
Setup a docker host that also happens to be listening on whatever ports:

ubuntu@execute-1460530807:~$ sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1204/sshd
tcp6       0      0 :::42069                :::*                    LISTEN      5017/node
tcp6       0      0 :::22                   :::*                    LISTEN      1204/sshd

Make sure that ICC is OFF:

ubuntu@execute-1460530807:~$ cat /etc/default/docker
DOCKER_OPTS="-s overlay --icc=false --log-driver=none"

Try to run a container that scans local available ports:

ubuntu@execute-1460530807:~$ docker run --rm -t us.gcr.io/coderpad-1189/coderpad:base /bin/bash -c "nc -vz 172.17.0.1 1-65535 2>&1 | grep succeeded"
Connection to 172.17.0.1 22 port [tcp/ssh] succeeded!
Connection to 172.17.0.1 42069 port [tcp/*] succeeded!

Describe the results you received:

The container was able to see and interact with the host over the local bridge.

Describe the results you expected:

https://github.com/docker/docker/blob/master/docs/userguide/networking/default_network/container-communication.md states:

It is a strategic question whether to leave --icc=true or change it to --icc=false so that iptables will protect other containers -- and the main host -- from having arbitrary ports probed or accessed by a container that gets compromised.

which suggests that this behavior should not occur with ICC disabled.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Apr 13, 2016

/cc @aboch

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Apr 17, 2016

ping

@aboch

This comment has been minimized.

Copy link
Contributor

@aboch aboch commented Apr 18, 2016

@vincentwoo Thanks for the report. I confirm currently there is a divergence between documentation and docker behavior. So I went back to docker 1.6.2 to see if that statement was respected. My finding was:

  • If the container tries to reach the exposed port via the docker0 bridge, the connection is in fact blocked
  • But if it tries against any other L3 interface on the host, the connection is allowed

So it looks like that statement was not very accurate. We need to decide what the expected behavior should be and make it consistent.

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Apr 20, 2016

Yeah, as a stopgap I added iptables -A INPUT -d 172.17.0.0/16 -i docker0 -j DROP to my systems but this is obviously brittle. I would love it if Docker behaved as suggested in the docs. I think anyone who runs icc = false would almost certainly want communication to the host to be blocked as well.

@aboch

This comment has been minimized.

Copy link
Contributor

@aboch aboch commented Apr 20, 2016

@vincentwoo I am still not sure about this.

Further in the docs, it says a way to allow two containers to communicate when --icc=false is to --link them. The example shows the two iptables rule which will allow the inter-communication as a result of linking. And these two rules are about the internal L4 exposed port for the containers IP addresses. IOW, to me this says there was an assumption that the two containers would access each other services via the original exposed ports on the containers IP, and not via third routing interfaces. In fact, once two containers are linked their etc/hosts file is populated accordingly so that direct resolution can happen, so that they can reach each other directly via their own IP + exposed ports.

I am just trying to understand better the context in which the statement was written.

It therefore appears to me that the blockage of accessing the container ports via host mapped port on the bridge L3 interface just came as for free by the way other rules were implemented and not by design.

Given this, -- and the main host -- should not have been mentioned in the docs, because that was not true for access via any host L3 interface besides the docker0.

My advice is to fix the documentation.

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Apr 20, 2016

That's totally fine as an outcome, but regardless of the environment in which the initial statement was written I think there's a clear practical need serviced by "block containers from accessing the host". Historically users who want this would probably just turn off ICC and call it a day. If we decided to remove that functionality from ICC, can Docker either

a) provide an option to disable container-to-host networking
b) document how that might be achieved stably on your own?

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Apr 20, 2016

@vincentwoo perhaps you're looking for the --internal option for networks #19276

docker network create --internal private-network

docker run -d --net=private-network --name web nginx

docker run -it --rm --net=private-network ubuntu

Inside that container; contacting the outside world fails:

root@eb7b58ee4a9f:/# ping -c1 google.com
ping: unknown host google.com

But communication with other containers on the same network is allowed;

root@eb7b58ee4a9f:/# ping -c1 web
PING web (172.21.0.2) 56(84) bytes of data.
64 bytes from web.private-network (172.21.0.2): icmp_seq=1 ttl=64 time=0.080 ms

--- web ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.080/0.080/0.080/0.000 ms
@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Apr 20, 2016

No, I want my containers to be able to contact the world, just not each other or the host.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Apr 20, 2016

If each container is in its own network, they should only be able to connect to containers on the same network, or containers that publish a port (so are publicly accessible from outside the docker host). Given that those port are already publicly accessible, I'm not sure if changing this brings much extra

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Apr 20, 2016

The host may have a port open on localhost, but be behind a firewall (the default on cloud providers). A potentially malicious actor in a container gets past the firewall by virtue of running on the same host. In so much as icc=false is a security measure, I (and other users) almost certainly apply it as a measure of protection.

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Jun 5, 2016

This has a 1.11 label but I don't think it was addressed.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jun 6, 2016

@vincentwoo the "version/1.11" label is added based on the version it was originally reported with

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Jun 7, 2016

Okay, that's fair. Are there plans to address this?

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jun 7, 2016

I don't know if this is being worked on at the moment (@aboch may know)

@andreisoare

This comment has been minimized.

Copy link

@andreisoare andreisoare commented Jun 17, 2016

I would also love to have a way to block connections from the container to the host, but allow the container to talk with the outside world, without having to use iptables. I was expecting icc=false to do this.

@xeor

This comment has been minimized.

Copy link

@xeor xeor commented Sep 22, 2016

There should really be an option for this, something like com.docker.network.bridge.enable_hc for host communication. The containers should not be able to talk to the ports the docker daemon server is listening on as default imh.

My fix is to block it all using iptables, then having 1 network dedicated to this for those few odd cases when you want it.

docker network create --subnet=192.168.0.0/24 --gateway=192.168.0.1 --ip-range=192.168.0.0/25 \
  -o "com.docker.network.bridge.enable_ip_masquerade"="false" \
  -o "com.docker.network.bridge.enable_icc"="false" \
  local-host-allow
iptables -I INPUT ! -s 192.168.0.0/24 -m addrtype --dst-type LOCAL -i docker0 -j DROP
@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Dec 13, 2016

Was this fixed by docker/libnetwork#1525? cc @aboch

@aboch

This comment has been minimized.

Copy link
Contributor

@aboch aboch commented Dec 13, 2016

@vincentwoo No, that fix is for honoring the ICC setting on --internal networks. I do not expect it to affect the behavior requested by this issue.

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Dec 13, 2016

Cool, thank you for answering my question.

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Jan 19, 2018

Hi, just want to check back in after a year ;)

Any plans to address?

@nathanielks

This comment has been minimized.

Copy link

@nathanielks nathanielks commented Jan 19, 2018

I'm curious myself!

@vincentwoo could you share what you've done to lock down your hosts? Do you have icc disabled and iptables enabled to allow linking, or have some other security measures in place?

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Jan 20, 2018

I don't use linking in production currently, so that's easier. I mostly have custom iptables rules on the host. On GCP I also have a rule to prevent containers from communicating with the GCP metadata server, because it authorizes requests by IP.

@raymond-ji

This comment has been minimized.

Copy link

@raymond-ji raymond-ji commented Feb 4, 2018

Following up if there's any plans to address this in Docker.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Feb 4, 2018

I'm not aware of someone working on this (@ddebroy @fcrisciani ?), but contributions are welcome

@brthor

This comment has been minimized.

Copy link

@brthor brthor commented Feb 20, 2018

It seems like this would be a common scenario causing security holes.

For example, a worker, running docker, awaits commands from some control server via an http server running on the host. All servers live inside the same VPC, so IP authentication is relied on for the HTTP service running on the worker.

The HTTP service will give varying levels of control over the docker daemon (depending on the application, but we can assume the http calls result in some kind of container being launched). In the worst case it's the HTTP api of dockerd itself.

Without host-container networking isolation, this host can be easily compromised by an unprivileged (even severely restricted) container.

I agree that some gesture in docker to restrict this is needed, but in the meantime some guidance on placing the proper iptables rules would be very helpful for the uninitiated (like me).

@vincentwoo Would you mind sharing your iptables rules you've crafted? Or the reasoning necessary to create them.

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Feb 20, 2018

Basically, dropping packets from the docker0 subnet to the host's local IP:

iptables -A INPUT -d 172.17.0.0/16 -i docker0 -j DROP

Of course, your specific addresses may vary.

@brthor

This comment has been minimized.

Copy link

@brthor brthor commented Feb 20, 2018

@xeor Your example doesn't seem to work at all without "com.docker.network.bridge.enable_ip_masquerade"="false" which I think most people are not going to want.

@vincentwoo Thanks!

I tried this out right now and it's working great. I'm creating networks for each container to work around #36355 , so in the generic sense:

iptables -A INPUT -d {network_subnet} -i {network_name} -j DROP

Where {network_name} is the bridge name found with brctl show or specified with com.docker.network.bridge.name during docker network create.

This seems like an easy enough rule to add (famous last words) when ICC is enabled if there's general consensus that it should be done (somewhat unclear in this thread).

Can someone point me at the implementation for the currently added rules? A search of iptables in this repository doesn't seem to find it (or I am missing it).

@vincentwoo

This comment has been minimized.

Copy link
Contributor Author

@vincentwoo vincentwoo commented Feb 20, 2018

I agree that that sounds like a good idea, but it'd be probably conditional on the network setup of the local machine. Some boxes may have multiple local LANs, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.