Cannot run container on new nodes

[cap10morgan]

I'm provisioning new Ubuntu 13.10 nodes in EC2 using knife-ec2. When I use docker::default to install Docker and then a docker_container resource after that, I get the error:

dial unix /var/run/docker.sock: no such file or directory

But when I SSH into the machine, that file is there and docker is running.

Here's the command that it attempts to run the container with:

docker run -cidfile /var/run/registry.cid -d -e SETTINGS_FLAVOR=prod -name registry -p 5000:5000 -v /mnt/docker-registry-storage:/tmp/registry -v /mnt/docker-registry-config:/docker-registry/config registry

When I SSH in immediately after the first Chef run, that command succeeds.

Any ideas?

[bflad]

Could you paste or Gist your full Chef run log? I'd also like to see your docker_image and docker_container resources please. Thanks!

[cap10morgan]

Here's the Chef run log (w/ a few bits redacted for security reasons): https://gist.github.com/cap10morgan/175c4fb1fdfde8768f43

docker_image resource:

docker_image 'registry'

docker_container resource:

docker_container 'registry' do
  detach true
  port '5000:5000'
  env ['SETTINGS_FLAVOR=prod']
  volume ['/mnt/docker-registry-storage:/tmp/registry', '/mnt/docker-registry-config:/docker-registry/config']
end

[bflad]

Awesome, thank you very much. I'll look deeper and see what I can't find.

On Tue, Jan 28, 2014 at 12:45 PM, Wes Morgan notifications@github.comwrote:

Here's the Chef run log (w/ a few bits redacted for security reasons):
https://gist.github.com/cap10morgan/175c4fb1fdfde8768f43

docker_image resource:
docker_image 'registry'

docker_container resource:
docker_container 'registry' do
detach true
port '5000:5000'
env ['SETTINGS_FLAVOR=prod']
volume ['/mnt/docker-registry-storage:/tmp/registry',
'/mnt/docker-registry-config:/docker-registry/config']
end

Reply to this email directly or view it on GitHubhttps://github.com//issues/49#issuecomment-33503791
.

Brian Flad
http://about.me/bflad

[cap10morgan]

Have you discovered anything on this? Is there anything else I can provide or test that might help?

[bflad]

I was going through some of my other cookbook backlogs over the weekend.
Hopefully can focus on this tonight.

On Mon, Feb 3, 2014 at 12:15 PM, Wes Morgan notifications@github.comwrote:

Have you discovered anything on this? Is there anything else I can provide
or test that might help?

Reply to this email directly or view it on GitHubhttps://github.com//issues/49#issuecomment-33976788
.

Brian Flad
http://about.me/bflad

[cap10morgan]

Interestingly, deploying all this to a c3.2xlarge HVM instance seems to have fixed the problem (at least everything worked on the first Chef run the first time I tried it). Before I was using c1.xlarge instances with instance-store AMIs (both official Ubuntu 13.10 images).

[bflad]

Okay this issue from what I can tell is two-fold:

• AWS instances intermittently being slow (to be expected)
• Cookbook not ensuring docker daemon is fully running

Basically, the docker daemon usually "starts" fairly quickly according to Upstart (and Chef), but it can take a little bit for the socket to be opened, especially on slow instances. You can test this on any t1.micro such as ami-7e64074e (us-west-2 Ubuntu 13.10 64-bit): stop docker && rm -f /var/log/upstart/docker.log && date && start docker && tail -f /var/log/upstart/docker.log check out the difference between the date command and when it reads Listening for HTTP on /var/run/docker.sock (unix).

In fact, on my t1.micro instance right now it took 14 minutes...

# stop docker && rm -f /var/log/upstart/docker.log && date && start docker && tail -f /var/log/upstart/docker.log
docker stop/waiting
Tue Feb  4 04:07:12 UTC 2014
docker start/running, process 4215
[/var/lib/docker|2c04b0d0] +job initapi()
[/var/lib/docker|2c04b0d0.initapi()] Creating server
Loading containers: : done.
2014/02/04 04:21:12 WARNING: Your kernel does not support cgroup swap limit.
[/var/lib/docker|2c04b0d0.initapi()] Creating pidfile
[/var/lib/docker|2c04b0d0.initapi()] Setting up signal traps
[/var/lib/docker|2c04b0d0] Register(export) (handlers=map[initapi:0x4a5530 version:0x4c18c0])
[/var/lib/docker|2c04b0d0] Register(create) (handlers=map[initapi:0x4a5530 version:0x4c18c0 export:0x4c72c0])
[/var/lib/docker|2c04b0d0] Register(stop) (handlers=map[initapi:0x4a5530 version:0x4c18c0 export:0x4c72c0 create:0x4c7300])
[/var/lib/docker|2c04b0d0] Register(start) (handlers=map[initapi:0x4a5530 version:0x4c18c0 export:0x4c72c0 create:0x4c7300 stop:0x4c7340])
[/var/lib/docker|2c04b0d0] Register(kill) (handlers=map[initapi:0x4a5530 version:0x4c18c0 export:0x4c72c0 create:0x4c7300 stop:0x4c7340 start:0x4c7380])
[/var/lib/docker|2c04b0d0] Register(serveapi) (handlers=map[initapi:0x4a5530 version:0x4c18c0 export:0x4c72c0 create:0x4c7300 stop:0x4c7340 start:0x4c7380 kill:0x4c73c0])
[/var/lib/docker|2c04b0d0] Register(wait) (handlers=map[initapi:0x4a5530 version:0x4c18c0 export:0x4c72c0 create:0x4c7300 stop:0x4c7340 start:0x4c7380 kill:0x4c73c0 serveapi:0x4c7400])
[/var/lib/docker|2c04b0d0] Register(tag) (handlers=map[version:0x4c18c0 create:0x4c7300 stop:0x4c7340 start:0x4c7380 wait:0x4c7440 initapi:0x4a5530 export:0x4c72c0 kill:0x4c73c0 serveapi:0x4c7400])
[/var/lib/docker|2c04b0d0] Register(resize) (handlers=map[version:0x4c18c0 create:0x4c7300 stop:0x4c7340 start:0x4c7380 wait:0x4c7440 initapi:0x4a5530 export:0x4c72c0 kill:0x4c73c0 serveapi:0x4c7400 tag:0x4c7480])
[/var/lib/docker|2c04b0d0] Register(commit) (handlers=map[version:0x4c18c0 create:0x4c7300 stop:0x4c7340 start:0x4c7380 wait:0x4c7440 resize:0x4c74c0 initapi:0x4a5530 export:0x4c72c0 kill:0x4c73c0 serveapi:0x4c7400 tag:0x4c7480])
[/var/lib/docker|2c04b0d0] Register(info) (handlers=map[version:0x4c18c0 create:0x4c7300 stop:0x4c7340 start:0x4c7380 wait:0x4c7440 resize:0x4c74c0 initapi:0x4a5530 export:0x4c72c0 kill:0x4c73c0 serveapi:0x4c7400 tag:0x4c7480 commit:0x4c7500])
[/var/lib/docker|2c04b0d0] -job initapi() = OK (0)
[/var/lib/docker|2c04b0d0] +job serveapi(unix:///var/run/docker.sock)
2014/02/04 04:21:12 Listening for HTTP on /var/run/docker.sock (unix)

An extreme example, but hardly desirable.

So what to do?

• @jcrobak has an excellent feature he's working on: jcrobak@85d3f25 that hopefully can make its way upstream. It'll fail Chef runs if the daemon timeout is reached when running any docker command.
• Don't depend on consistency in AWS

If you have other ideas, I'm all ears.

[bflad]

Fix in 0.30.0 which will be released tonight.