weave cannot delete container

[Ozzyboshi]

When i try to launch weave in my swarm node i get this error
root@swarm-node-04:~# weave launch
Error deleting container: Error response from daemon: Unable to remove filesystem for 1755b23b10866234fc320bf8739e6343676426ab24be5032bb325e4179099c66: remove /var/lib/docker/containers/1755b23b10866234fc320bf8739e6343676426ab24be5032bb325e4179099c66/shm: device or resource busy
This node was build on a digitalocean vps and configured with docker-machine generic driver, for example

docker-machine create --driver generic --generic-ip-address XXXX --generic-ssh-user root --swarm --swarm-discovery token://$sid swarm-node-04

Any ideas?

[rade]

What version of weave are you running?
Also, please post the output of docker info.

[Ozzyboshi]

Thanks for your help, now I am going to give you step by step instruction how to reproduce this problem:

1. go to digitalocean and activate a new ubuntu 14.04 droplet (I think the same thing occurs with a regular ubuntu desktop machine also)

2. install docker swarm node with docker-machine generic driver (you must be able to ssh with a key rather than a password) :

   sid="yourswarktockenhere" && docker-machine create --driver generic --generic-ip-address vpspubliciphere --generic-ssh-user root  --swarm  --swarm-discovery token://$sid swarm-node-04
   

   this takes a while, at the end i expect docker installed with a running swarm agent container.

3. ssh into the vps and run

   curl -L git.io/weave -o /usr/local/bin/weave
   chmod a+x /usr/local/bin/weave
   

4. run weave launch, it will download the latest weave images and run them but something goes wrong...

   root@swarm-node-04:~# weave launch
   Unable to find image 'weaveworks/weaveexec:1.2.1' locally
   1.2.1: Pulling from weaveworks/weaveexec
   8a648f689ddb: Pull complete 
   32cb7ec428b9: Pull complete 
   442846c7a265: Pull complete 
   b6612c070bd9: Pull complete 
   6ecb73994984: Pull complete 
   98c69b3e433b: Pull complete 
   350505f52d83: Pull complete 
   b7b1df8af663: Pull complete 
   75425d4f82d6: Pull complete 
   3d6b4db0af70: Pull complete 
   4e9d7ba10157: Pull complete 
   b1bdb47f447f: Pull complete 
   183508e67c80: Pull complete 
   Digest: sha256:1d9c31fecc9c669613139c567d60075fb985b5e5ee7eb7254c196828a6f6e8a3
   Status: Downloaded newer image for weaveworks/weaveexec:1.2.1
   Unable to find image 'weaveworks/weave:1.2.1' locally
   1.2.1: Pulling from weaveworks/weave
   Digest: sha256:c08dfd7f0453ef2dc307abbd970318b4679d832f88a5fed63f08449c1c91fa28
   Status: Downloaded newer image for weaveworks/weave:1.2.1
   Error deleting container: Error response from daemon: Unable to remove filesystem for             02b7f18c5663a7626e457080a3755f17d459081dea197d48ce65b9ba1c595a5b: remove       /var/lib/docker/containers/02b7f18c5663a7626e457080a3755f17d459081dea197d48ce65b9ba1c595a5b/shm: device or resource busy
   

Docker info output:

 root@swarm-node-04:~# docker info
 Containers: 4
 Images: 26
Server Version: 1.9.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 32
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-57-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 1
Total Memory: 490 MiB
Name: swarm-node-04
ID: KD4B:UBEY:DOVI:BQKG:I5JK:U23N:OVG2:USRY:CHUP:3JCK:WLZP:6RXT
WARNING: No swap limit support
Labels:
provider=generic

Docker version:

root@swarm-node-04:~# docker version
Client:
Version:      1.9.0
API version:  1.21
Go version:   go1.4.2
Git commit:   76d6bc9
Built:        Tue Nov  3 17:43:42 UTC 2015
OS/Arch:      linux/amd64

Server:
Version:      1.9.0
API version:  1.21
Go version:   go1.4.2
Git commit:   76d6bc9
Built:        Tue Nov  3 17:43:42 UTC 2015
OS/Arch:      linux/amd64

Weave version:

root@swarm-node-04:~# weave version
weave script 1.2.1
weave router 1.2.1
weave proxy  1.2.1

Even if weave launch exited with an error I see 2 weave running containers, one named weave and the other is weaveproxy.
I expect to communicate with weave-proxy though 12375/tcp but it seems that weave-proxy is not bound to this port but it should, maybe he's not bound because of the above error.
I see that there is also a weave dead container, it's very strange...

root@swarm-node-04:~# docker ps -a
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS               NAMES
06570dc6131f        weaveworks/weaveexec:1.2.1   "/home/weave/weavepro"   22 minutes ago      Up 22 minutes                           weaveproxy
793a2ba56135        weaveworks/weave:1.2.1       "/home/weave/weaver -"   23 minutes ago      Up 23 minutes                           weave
02b7f18c5663        weaveworks/weaveexec:1.2.1   "/home/weave/sigproxy"   23 minutes ago      Dead                                    
a76accb70b3f        swarm:latest                 "/swarm join --advert"   28 minutes ago      Up 28 minutes       2375/tcp            swarm-agent

All this stuff is because I tried to follow this tutorial by @inercia : http://blog.weave.works/2015/11/06/weave-discovery-and-docker-swarm/#more-1759
I am happy to provide more debug info if needed.

[rade]

We have seen similar errors in container deletion on docker 1.9.0. It's hard to pin down though.

One thing that did appear to cure the problem for me was switching docker's storage driver from aufs to overlay. That needs a more recent kernel than stock ubuntu 14.04 though.

[llarsson]

I am running a setup similar to the one described in Part 3 of the official guide and run in to the same issue.

Are there Linux distros that are known to work without this issue? I would of course much rather prefer to have my docker-machine created instances be on some kind of a long-term support type distro version than not. CentOS, Debian, whatever would also be fine for me.

[rade]

14.04 with a 3.19 kernel - which you can get through the LTS Enabledment Stack - works just fine.

[bboreham]

We realised the part of weave that triggers this error doesn't need access to that particular directory, so this will be fixed in the next point release.

[Ozzyboshi]

It seems that this problem is solved with a 3.19 kernel , however Ubuntu 14.04 is still the latest LTS and it ships with 3.13. I don't know if it is the case to fix it or wait until the next Ubuntu LTS release due next April, it' s up to you. I leave this issue open but if you want to close it I am OK