New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unable to link containers with --iptables=false #12701

Closed
fl33t opened this Issue Apr 23, 2015 · 13 comments

Comments

Projects
None yet
8 participants
@fl33t

fl33t commented Apr 23, 2015

Description

Docker ehazlett/logstash fails to complete starting due to the error:

Apr 23 10:50:11 myhost kernel: [ 2190.206988] aufs au_opts_verify:1570:docker[2926]: dirperm1 breaks the protection by the permission bits on the lower branch

The error occurred only after updating to version 1.6.0 and restarting the docker. The docker used is a simple log monitor service.

uname -a:
Linux myhost 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt9-2~bpo70+1 (2015-04-21) x86_64 GNU/Linux

docker version:

Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64

docker info:

Containers: 6
Images: 66
Storage Driver: aufs
 Root Dir: /srv/docker/aufs
 Backing Filesystem: extfs
 Dirs: 82
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.16.0-0.bpo.4-amd64
Operating System: Debian GNU/Linux 7 (wheezy)
CPUs: 1
Total Memory: 1.963 GiB
Name: myhost
ID: XYNG:SVK2:IFUD:7SRT:JUFH:QCRZ:MVNC:YYQE:KELN:P3GC:JRNI:LPMS
WARNING: No memory limit support
WARNING: No swap limit support

Environment Details

  • KVM virtual machine

Steps to Reproduce

  • Start the elasticsearch docker
    • docker run -d -it -p 9200:9200 -p 9300:9300 --name es ehazlett/elasticsearch
  • Start the logstash docker
    • docker run --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch ehazlett/logstash -f /etc/logstash.conf.sample

Actual Results

stdout:

root@myhost:/home/myuser# docker run -d --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch --name logstash ehazlett/logstash
b5ffe53543ceb05415fcbdf749197ae6baf5a98b25e30b51f2ed85e6ed99196a
FATA[0000] Error response from daemon: Cannot start container b5ffe53543ceb05415fcbdf749197ae6baf5a98b25e30b51f2ed85e6ed99196a:  (exit status 1) 

Syslog output:
root@myhost:/home/lattice# tail -n 3 /var/log/syslog Apr 23 10:50:11 myhost kernel: [ 2189.967344] aufs au_opts_verify:1570:docker[2926]: dirperm1 breaks the protection by the permission bits on the lower branch Apr 23 10:50:11 myhost kernel: [ 2190.144856] aufs au_opts_verify:1570:docker[2926]: dirperm1 breaks the protection by the permission bits on the lower branch Apr 23 10:50:11 myhost kernel: [ 2190.206988] aufs au_opts_verify:1570:docker[2926]: dirperm1 breaks the protection by the permission bits on the lower branch

docker log output:

root@myhost:/home/myuser# tail -n 2000 /var/log/docker.log | grep 10:50
time="2015-04-23T10:50:11-06:00" level=info msg="POST /v1.18/containers/create?name=logstash" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job create(logstash)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job log(create, 4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0, ehazlett/logstash:latest)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job log(create, 4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0, ehazlett/logstash:latest) = OK (0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job create(logstash) = OK (0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="POST /v1.18/containers/4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0/start" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job start(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job allocate_interface(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job allocate_interface(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0) = OK (0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job allocate_port(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job allocate_port(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0) = OK (0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job allocate_port(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job allocate_port(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0) = OK (0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job link(-A)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job link(-A) = ERR (1)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job link(-D)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job link(-D) = OK (0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job release_interface(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job release_interface(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0) = OK (0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job log(die, 4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0, ehazlett/logstash:latest)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job log(die, 4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0, ehazlett/logstash:latest) = OK (0)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job start(4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0) = ERR (1)" 
time="2015-04-23T10:50:11-06:00" level=error msg="Handler for POST /containers/{name:.*}/start returned error: Cannot start container 4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0:  (exit status 1)" 
time="2015-04-23T10:50:11-06:00" level=error msg="HTTP Error: statusCode=500 Cannot start container 4c23291092e74d7c6dac293dde0b3b52689662f06b2fecf2475e9edb851baad0:  (exit status 1)" 

Expected Results

  • Docker should start without error
@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83
Contributor

cpuguy83 commented Apr 23, 2015

@dqminh

This comment has been minimized.

Show comment
Hide comment
@dqminh

dqminh Apr 23, 2015

Contributor

@fl33t is the command to run logstash container correct ? When i tried it in foreground mode, the output is:

~ ❯ docker run --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch --name logstash ehazlett/logstash
No command given
Usage: logstash <command> [command args]
Run a command with the --help flag to see the arguments.
For example: logstash agent --help

Available commands:
  agent - runs the logstash agent
  version - emits version info about this logstash
  web - runs the logstash web ui (called Kibana)
  rspec - runs tests

I dont think aufs will prevent you from starting the container. dirperm1 breaks the protection by the permission bits on the lower branch is only a warning produced by aufs, it should not affect container's start.

Contributor

dqminh commented Apr 23, 2015

@fl33t is the command to run logstash container correct ? When i tried it in foreground mode, the output is:

~ ❯ docker run --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch --name logstash ehazlett/logstash
No command given
Usage: logstash <command> [command args]
Run a command with the --help flag to see the arguments.
For example: logstash agent --help

Available commands:
  agent - runs the logstash agent
  version - emits version info about this logstash
  web - runs the logstash web ui (called Kibana)
  rspec - runs tests

I dont think aufs will prevent you from starting the container. dirperm1 breaks the protection by the permission bits on the lower branch is only a warning produced by aufs, it should not affect container's start.

@fl33t

This comment has been minimized.

Show comment
Hide comment
@fl33t

fl33t Apr 23, 2015

@dqminh I corrected the logstash startup command to the correct values:

  • docker run --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch ehazlett/logstash -f /etc/logstash.conf.sample

The command will work on 1.5.0, but not on 1.6.0.

fl33t commented Apr 23, 2015

@dqminh I corrected the logstash startup command to the correct values:

  • docker run --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch ehazlett/logstash -f /etc/logstash.conf.sample

The command will work on 1.5.0, but not on 1.6.0.

@dqminh

This comment has been minimized.

Show comment
Hide comment
@dqminh

dqminh Apr 23, 2015

Contributor

@fl33t that's weird, i just tried and i can run it without any issues

docker run --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch ehazlett/logstash -f /etc/logstash.conf.sample

Using milestone 2 input plugin 'tcp'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones {:level=>:warn}
Using milestone 2 input plugin 'udp'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones {:level=>:warn}
Using milestone 1 filter plugin 'syslog_pri'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones {:level=>:warn}

^CInterrupt received. Shutting down the pipeline. {:level=>:warn}
UDP listener died {:exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/opt/logstash/lib/logstash/inputs/udp.rb:80:in `udp_listener'", "org/jruby/RubyKernel.java:1521:in `loop'", "/opt/logstash/lib/logstash/inputs/udp.rb:78:in `udp_listener'", "/opt/logstash/lib/logstash/inputs/udp.rb:50:in `run'", "/opt/logstash/lib/logstash/pipeline.rb:163:in `inputworker'", "/opt/logstash/lib/logstash/pipeline.rb:157:in `start_input'"], :level=>:warn}

So in your case, the command failed to run ? What's the output of docker logs container-id ?

Contributor

dqminh commented Apr 23, 2015

@fl33t that's weird, i just tried and i can run it without any issues

docker run --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch ehazlett/logstash -f /etc/logstash.conf.sample

Using milestone 2 input plugin 'tcp'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones {:level=>:warn}
Using milestone 2 input plugin 'udp'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones {:level=>:warn}
Using milestone 1 filter plugin 'syslog_pri'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones {:level=>:warn}

^CInterrupt received. Shutting down the pipeline. {:level=>:warn}
UDP listener died {:exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/opt/logstash/lib/logstash/inputs/udp.rb:80:in `udp_listener'", "org/jruby/RubyKernel.java:1521:in `loop'", "/opt/logstash/lib/logstash/inputs/udp.rb:78:in `udp_listener'", "/opt/logstash/lib/logstash/inputs/udp.rb:50:in `run'", "/opt/logstash/lib/logstash/pipeline.rb:163:in `inputworker'", "/opt/logstash/lib/logstash/pipeline.rb:157:in `start_input'"], :level=>:warn}

So in your case, the command failed to run ? What's the output of docker logs container-id ?

@dqminh

This comment has been minimized.

Show comment
Hide comment
@dqminh

dqminh Apr 23, 2015

Contributor

@cpuguy83 @unclejack That's being said, maybe we should have some option to disable dirperm1 (i.e., --storage-opt aufs.mountopt=nodirperm1 for user that doesn't have to deal with #783 , and doesnt want to have aufs warning in their kernel logs ( sigh aufs ... ). I think i still have the patch hanging around somewhere.

Contributor

dqminh commented Apr 23, 2015

@cpuguy83 @unclejack That's being said, maybe we should have some option to disable dirperm1 (i.e., --storage-opt aufs.mountopt=nodirperm1 for user that doesn't have to deal with #783 , and doesnt want to have aufs warning in their kernel logs ( sigh aufs ... ). I think i still have the patch hanging around somewhere.

@fl33t

This comment has been minimized.

Show comment
Hide comment
@fl33t

fl33t Apr 23, 2015

@dqminh It does fail to run and does not appear to have a STATUS in the list containers view. There does not appear to be any logs. I have provided the example run below:

myuser@myhost:~$ docker run --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch --name logstash ehazlett/logstash
FATA[0000] Error response from daemon: Cannot start container 2a752ea9eb7de9cbbd178ab3d767b2ea8ff848e583da19e567cd93487ab106a3:  (exit status 1) 
myuser@myhost:~$ docker logs 2a752ea9eb7de9cbbd178ab3d767b2ea8ff848e583da19e567cd93487ab106a3
myuser@myhost:~$ docker ps -a
CONTAINER ID        IMAGE                           COMMAND                CREATED             STATUS                   PORTS                                            NAMES
2a752ea9eb7d        ehazlett/logstash:latest        "/opt/logstash/bin/l   15 minutes ago                                                                                logstash            
01f7a09c67c7        ehazlett/elasticsearch:latest   "/opt/elasticsearch/   8 days ago          Up 5 hours               0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   es                  

fl33t commented Apr 23, 2015

@dqminh It does fail to run and does not appear to have a STATUS in the list containers view. There does not appear to be any logs. I have provided the example run below:

myuser@myhost:~$ docker run --restart="always" -it -p 5000:5000 -p 5000:5000/udp --link es:elasticsearch --name logstash ehazlett/logstash
FATA[0000] Error response from daemon: Cannot start container 2a752ea9eb7de9cbbd178ab3d767b2ea8ff848e583da19e567cd93487ab106a3:  (exit status 1) 
myuser@myhost:~$ docker logs 2a752ea9eb7de9cbbd178ab3d767b2ea8ff848e583da19e567cd93487ab106a3
myuser@myhost:~$ docker ps -a
CONTAINER ID        IMAGE                           COMMAND                CREATED             STATUS                   PORTS                                            NAMES
2a752ea9eb7d        ehazlett/logstash:latest        "/opt/logstash/bin/l   15 minutes ago                                                                                logstash            
01f7a09c67c7        ehazlett/elasticsearch:latest   "/opt/elasticsearch/   8 days ago          Up 5 hours               0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   es                  
@dqminh

This comment has been minimized.

Show comment
Hide comment
@dqminh

dqminh Apr 24, 2015

Contributor

@fl33t i think the problem here is not with aufs and dirperm1, but with linking containers. Somehow the container failed to link, based on your log

time="2015-04-23T10:50:11-06:00" level=info msg="+job link(-A)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job link(-A) = ERR (1)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job link(-D)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job link(-D) = OK (0)" 
Contributor

dqminh commented Apr 24, 2015

@fl33t i think the problem here is not with aufs and dirperm1, but with linking containers. Somehow the container failed to link, based on your log

time="2015-04-23T10:50:11-06:00" level=info msg="+job link(-A)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job link(-A) = ERR (1)" 
time="2015-04-23T10:50:11-06:00" level=info msg="+job link(-D)" 
time="2015-04-23T10:50:11-06:00" level=info msg="-job link(-D) = OK (0)" 
@airbillion

This comment has been minimized.

Show comment
Hide comment
@airbillion

airbillion Apr 26, 2015

I have the same issue trying to start my dockerui container. It failed to start and I have the same error displayed.
This is my run command:
docker run -d --name="dockerui-omv" --restart="always" --net="host" --privileged -v /var/run/docker.sock:/var/run/docker.sock dockerui/dockerui

I an not linking to another container and this only occurred after updating to 1.6.

Any solutions?

airbillion commented Apr 26, 2015

I have the same issue trying to start my dockerui container. It failed to start and I have the same error displayed.
This is my run command:
docker run -d --name="dockerui-omv" --restart="always" --net="host" --privileged -v /var/run/docker.sock:/var/run/docker.sock dockerui/dockerui

I an not linking to another container and this only occurred after updating to 1.6.

Any solutions?

@fl33t

This comment has been minimized.

Show comment
Hide comment
@fl33t

fl33t Apr 28, 2015

@dqminh I was able to identify the issue as stemming from iptables manipulation in container linking. The line below from the /var/log/docker.log shows the iptables command manipulation failing.

time="2015-04-27T19:09:27-06:00" level=info msg="+job link(-A)" 
iptables failed: iptables -t filter -A DOCKER -i docker0 -o docker0 -p tcp -s 172.17.0.20 -d 172.17.0.16 --dport 9200 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)
time="2015-04-27T19:09:27-06:00" level=info msg="-job link(-A) = ERR (1)" 
time="2015-04-27T19:09:27-06:00" level=info msg="+job link(-D)" 
time="2015-04-27T19:09:27-06:00" level=info msg="-job link(-D) = OK (0)" 

The cause for this iptables error is a missing chain named DOCKER. The DOCKER chain is missing as the line DOCKER_OPTS="-g /srv/docker --iptables=false" was placed in our /etc/default/docker to prevent iptables manipulation. However, it had the unintended consequence of preventing container linking. After creating the DOCKER chain, the containers will link properly.

sudo iptables -N DOCKER

Would it be appropriate to close the issue or relabel it or both?

fl33t commented Apr 28, 2015

@dqminh I was able to identify the issue as stemming from iptables manipulation in container linking. The line below from the /var/log/docker.log shows the iptables command manipulation failing.

time="2015-04-27T19:09:27-06:00" level=info msg="+job link(-A)" 
iptables failed: iptables -t filter -A DOCKER -i docker0 -o docker0 -p tcp -s 172.17.0.20 -d 172.17.0.16 --dport 9200 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)
time="2015-04-27T19:09:27-06:00" level=info msg="-job link(-A) = ERR (1)" 
time="2015-04-27T19:09:27-06:00" level=info msg="+job link(-D)" 
time="2015-04-27T19:09:27-06:00" level=info msg="-job link(-D) = OK (0)" 

The cause for this iptables error is a missing chain named DOCKER. The DOCKER chain is missing as the line DOCKER_OPTS="-g /srv/docker --iptables=false" was placed in our /etc/default/docker to prevent iptables manipulation. However, it had the unintended consequence of preventing container linking. After creating the DOCKER chain, the containers will link properly.

sudo iptables -N DOCKER

Would it be appropriate to close the issue or relabel it or both?

@fl33t fl33t changed the title from dirperm1 breaks the protection by the permission bits on the lower branch to unable to link containers with --iptables=false May 5, 2015

@fl33t fl33t closed this May 5, 2015

@thaJeztah thaJeztah added the Networking label May 5, 2015

@danielovalle

This comment has been minimized.

Show comment
Hide comment
@danielovalle

danielovalle May 16, 2015

@dqminh THANKS SO MUCH!!!

danielovalle commented May 16, 2015

@dqminh THANKS SO MUCH!!!

@iflederick

This comment has been minimized.

Show comment
Hide comment
@iflederick

iflederick Jun 3, 2015

Running into this problem, using docker 1.6 on CentOS 7.
The issue should be re-opened, because I don't want to pass "iptables=true" or create a DOCKER chain, and still want the ability to link containers?

iflederick commented Jun 3, 2015

Running into this problem, using docker 1.6 on CentOS 7.
The issue should be re-opened, because I don't want to pass "iptables=true" or create a DOCKER chain, and still want the ability to link containers?

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Jun 3, 2015

Contributor

@iflederick @dqminh @danielovalle this is addressed in 1.7.0. with the new libnetwork changes. we dont make use of iptables rules for links if --icc=true (the default case). But we will install iptables rules if --icc=false.

Since we cannot have --icc=false and --iptables=false rules are anyways, links should just work fine even with iptables=false.

Can you please try the 1.7.0-rc1 image and confirm ?

Contributor

mavenugo commented Jun 3, 2015

@iflederick @dqminh @danielovalle this is addressed in 1.7.0. with the new libnetwork changes. we dont make use of iptables rules for links if --icc=true (the default case). But we will install iptables rules if --icc=false.

Since we cannot have --icc=false and --iptables=false rules are anyways, links should just work fine even with iptables=false.

Can you please try the 1.7.0-rc1 image and confirm ?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jun 7, 2015

Member

For those willing to test; the current 1.7-rc2 can be found here #13528 (comment) and RPMs can be found here; #13528 (comment)

The obligatory warnings; These are release candidates so don't use them on critical data and be aware that downgrading to 1.6 after running 1.7 may not be possible in all cases.

Member

thaJeztah commented Jun 7, 2015

For those willing to test; the current 1.7-rc2 can be found here #13528 (comment) and RPMs can be found here; #13528 (comment)

The obligatory warnings; These are release candidates so don't use them on critical data and be aware that downgrading to 1.6 after running 1.7 may not be possible in all cases.

@mfojtik mfojtik referenced this issue Mar 13, 2016

Closed

docker issue #7977

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment