ip-forwarding broken in 1.13.0 due to #28257 fix

[AlmirKadric]

Description

#28257 introduced a fix which switched the default iptables FORWARDING policy to DROP. However existing setups rely on this to route traffic into the guest containers from the host.

I was able to work around the issue by disabling iptables in the daemon configuration. But this is probably unacceptable for most.

The only suggestions I can think of would be to either allow the configuration of default iptables FORWARDING policies via the dameon config or similar AND/OR allow the configuring of these policies per configured network (i.e. netwoks section in docker-compose files (I understand docker-compose is a separate project, but its what I'm using so dropped example for context)).

Steps to reproduce the issue:

1. Add route to configured network address range with eht0 or hvint0 as a Gateway
2. Try and ping a container IP address

Describe the results you received:
Unable to reach address

Describe the results you expected:
Successful ping

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

$ docker version
Client:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Wed Jan 18 16:20:26 2017
 OS/Arch:      windows/amd64

Server:
 Version:      1.13.0
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Wed Jan 18 16:20:26 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

$ docker info
Containers: 3
 Running: 3
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 1.13.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.4-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.934 GiB
Name: moby
ID: HEBW:ZJ2J:6FPC:H4O3:BASB:DYHE:BFZY:6GEN:XSEU:CNN6:4X7N:NV3Q
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 36
 Goroutines: 39
 System Time: 2017-01-21T08:16:31.8639268Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Windows 10, Docker for Windows

[justincormack]

cc @sanimej

[sanimej]

@AlmirKadric ip-forwarding is not broken. This flag is to control if docker should enable IP forwarding and it still does that. But what has changed is setting the default FORWARD policy to DROP. If you see the discussion in #14041 its clearly what the users wanted. Since the previous behavior was considered a vulnerability (ie: using a static route across hosts to reach containers on a different host) it was a conscious decision to change the default behavior.

If you were relying on the earlier behavior the work around I would suggest is to set the default FORWARD policy explicitly rather than relying on Docker doing it.

[AlmirKadric]

@sanimej thank you for your speedy response.
Regarding setting the default FORWARDING policy, If you look at my issue I mentioned that as a suggested fix. However how do you suggest I do that? Currently there is no docker daemon config option to setup firewall rules, and there is no clean way to ssh into the docker host instance on on Docker for Mac or Windows. Please do correct me if I am missing something??? Also keyword here is Docker for Windows or Mac not Docker Machine. So we cannot use docker-machine ssh commands.

Whilst I do understand this being a conscious decision, the community needs replacements for existing behaviours, we can't just condemn them.

EDIT: also just an FYI in case it becomes a solution, editing the daemon config is a pain, we need a way to make these changes via command line so that the process can be automated.

[AlmirKadric]

@sanimej any news on this?
any suggestions?

[adamaig]

Based on the discussion and issues this has raised in the community where it has broken functionality (see this pipeline issue) this is feeling like a botched feature release.

The iptables manipulation is under-documented and should be called out as a separate sub-topic in the docker-engine network guide--it seems a bit hidden in the "how containers communicate" sub-sub-section. Additionally, this feature shouldn't be buried far down the release notes, because it can break so many setups.

It really seems like it would have been preferable to make this opt-in functionality in this release, with well-advertised notices and discussion of appropriate workarounds on the different supported platforms.

[sanimej]

@AlmirKadric With Docker for Mac/Windows you can't directly SSH to the docker host. But you can run a privileged container, docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh and set the default policy to ALLOW if thats what you want. IMO that would be better than disabling iptables.

The reason for setting the default FORWARD policy to DROP is to disallow inadvertent access to local containers from other hosts in the network. Its not clear how this affects you in the case of Docker for Mac. Can you explain what exactly you are doing ?

[sanimej]

@adamaig Like I mentioned the earlier behavior was considered a security hole and that is the reason for changing the default behavior.

Note that when you upgrade to 1.13 if the host already has ip_forward enabled (it should be case if you were already using docker with ip_forward enabled) this change will not be done. Its done only if docker enables ip_forward. It was done this way to avoid breaking working setups that rely on the previous behavior. I will look into the documentation aspect and add an explicit note about this.

If your setup needs the previous behavior you can configure the default FORWARD policy to ACCEPT.

[adamaig]

@sanimej my point is that if by default this is the iptables config in a distribution (e.g., ubuntu-16.04 and fedora-24)...

root@vagrant:/home/vagrant# iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

and the docker 1.13 release changes that behavior by default, then I think there is a problem with the approach. The problem is that someone maybe leveraging that lack of necessary configuration for non-docker work, and then rather than installing an iptables config that is externally accessible, users now have to research and discover the right load order and tools to revert this override.

So, the documentation needs to cover how to undo what docker is doing to the system. Fundamentally changing the rule default, and not giving clear instruction on how to override the docker daemon's changes is frustrating and wastes user's time. The company needs to advertise a "contract" with the user for how they can configure and manage their systems without it suddenly breaking without deprecation warnings.

[AlmirKadric]

@sanimej I forgot to get back on here and let you know that your suggestion worked
I have integrated that into my automation scripts and haven't had any issues since
thank you very much for your support 👍