New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOCKER iptables chain problem when used with geerlingguy.firewall #21

Closed
ps-aux opened this Issue Aug 7, 2017 · 8 comments

Comments

Projects
None yet
5 participants
@ps-aux
Copy link

ps-aux commented Aug 7, 2017

I have a simple playbook

- hosts: virt
  vars:
    firewall_allowed_tcp_ports:
    - "22"
  roles:
    - firewall
    - docker

After successfull completion running docker run -p 1234:1234 hello-world results in:

docker: Error response from daemon: driver failed programming external connectivity on endpoint amazing_mestorf (3dcbb599298bbd13e3769ef93d5659c25185f143f819250dc9a67f80caefe31c): (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1234 -j ACCEPT: iptables: No chain/target/match by that name. (exit status 1)).

Workaround is to restart the Docker service

@PhilThurston

This comment has been minimized.

Copy link

PhilThurston commented Aug 17, 2017

Docker modifies Iptables to fit its needs at the time that a container is run. I'm not sure about that other role since we use our own solution, but we manage our own firewall just like you seem to be doing. My guess at the issue is that docker is messing with some of your firewall rules or your firewall isn't allowing docker to make those rules. This is what we do to solve these conflicts.

First, we configure docker to not touch IPtables at all you can do this with making/editing the /etc/docker/daemon.json (the ability to edit this is pending in a pull request) and adding this line:

{
  "iptables": false
}

Next you will want to add some iptables commands to your scripts to do the work that docker would have.

/sbin/iptables -N DOCKER

# Masquerade outbound connections from containers
/sbin/iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

# Accept established connections to the docker containers
/sbin/iptables -t filter -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Allow docker containers to communicate with themselves & outside world
/sbin/iptables -t filter -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
/sbin/iptables -t filter -A FORWARD -i docker0 -o docker0 -j ACCEPT

Doing this will set up the rules that docker will use to connect different containers and allow them to communicate with the outside world. We use this on several production servers without issue.

Good luck! 😃

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Aug 18, 2017

Interesting... I might need to do this. Currently, I have a bit of a hack to just do a reboot now on each instance where Docker is installed, so that the iptables configuration is all set up correctly (a restart fixes the OP's issue).

But making this role integrate better with the geerlingguy.firewall role would be a good improvement in terms of first-time setup on a more secure server!

@zzn01

This comment has been minimized.

Copy link

zzn01 commented Feb 11, 2018

any update of this issue?

@noplanman

This comment has been minimized.

Copy link

noplanman commented Apr 2, 2018

Still no update on this?

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented May 2, 2018

Don't worry, I'm still thinking about the best fix here. Testing some things now because I'm tired of my first build failing for many of my servers.

For now, the simplest fix (until this issue is closed) is to do one of the following:

  • Restart the Docker daemon. After restarting Docker, everything should work fine.
  • Reboot the server. After reboot, everything should work fine.

Also, the full error message on first start, for posterity:

TASK [Bring up the Docker containers.] *****************************************
fatal: [192.168.44.3]: FAILED! => {
  "changed": true,
  "cmd": [
    "docker-compose",
    "up",
    "-d",
    "--remove-orphans"
  ],
  "delta": "0:00:00.446252",
  "end": "2018-05-01 18:51:02.695903",
  "msg": "non-zero return code",
  "rc": 1,
  "start": "2018-05-01 18:51:02.249651",
  "stderr": "Creating network \"cores_default\" with the default driver
Failed to program FILTER chain: iptables failed: iptables --wait -I FORWARD -o br-ef94b61e2c8a -j DOCKER: iptables v1.6.0: Couldn't load target `DOCKER':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.\n (exit status 2)",
  "stderr_lines": [
    "Creating network \"cores_default\" with the default driver",
    "Failed to program FILTER chain: iptables failed: iptables --wait -I FORWARD -o br-ef94b61e2c8a -j DOCKER: iptables v1.6.0: Couldn't load target `DOCKER':No such file or directory",
    "",
    "Try `iptables -h' or 'iptables --help' for more information.",
    " (exit status 2)"
  ],
  "stdout": "",
  "stdout_lines": [
    
  ]
}

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented May 2, 2018

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented May 2, 2018

Fix is in 07e05ef, and that's probably the simplest solution while keeping your firewall rules and configuration somewhat sane—basically make sure you run the geerlingguy.firewall role (or manage whatever firewall changes you need to manage) before this role, and then this role will run a meta: flush_handlers to restart Docker after restarting the firewall (if necessary), thus ensuring the Docker iptables rules are intact after this role's work is complete.

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented May 2, 2018

The fix will be in 2.2.0.

kakawait added a commit to kakawait/ansible-role-docker that referenced this issue Jun 5, 2018

Merge remote-tracking branch 'upstream/master'
* upstream/master: (23 commits)
  Fixes geerlingguy#42: Allow control over docker_service state.
  Update docker-compose version to 1.21.2
  Fix deprecation warning
  PR geerlingguy#26 follow-up: Docs tidy.
  PR geerlingguy#43 follow-up: Align order of vars in docs with defaults.
  Docker repo doesn't yet support Ubuntu 18.04 Bionic, it seems.
  PR geerlingguy#47 follow-up: Add tests for Ubuntu 18.04 Bionic, Fedora 27, and fix boolean case.
  Fixes geerlingguy#21: DOCKER iptables chain problem when used with geerlingguy.firewall.
  Issue geerlingguy#54: Update Docker Compose default version to the latest version. For real this time.
  Issue geerlingguy#54: Update Docker Compose default version to the latest version.
  Make ignoring repository key error optional
  Support fedora
  Changed 'include' to 'include_tasks' due to deprecations in Ansible >2.4
  Introduced CPU architecture switch for apt source definition
  'include' for tasks has been deprecated
  Update documentation
  Bump docker_compose_version to 1.16.1.
  Change include_task to depreciated include due to what looks like incompatibility with test harness
  Fix docker group to append group rather than set it
  Fix typo
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment