failed to create container for some docker app after successfully staged

[libnux]

I tried to push a very simple docker image, but failed with message
"
2015-12-08T16:47:36.86+0800 [CELL/0] OUT Creating container
2015-12-08T16:47:56.93+0800 [CELL/0] ERR Failed to create container
"
I checked the garden log, and found some errors:

{"timestamp":"1449564439.654299736","source":"garden-linux","message":"garden-linux.container.start.command.failed","log_level":2,"data":{"argv":["/var/vcap/data/garden/depot/7f24gt0oua
j/start.sh"],"error":"exit status 2","exit-status":2,"handle":"c3ae6bcd-5825-4510-81e2-5e13089c868d-5181b8b2-aa86-4a3a-8e8d-54ccfc1f6fc1-0835adbc-f0ce-4d89-4154-cba0d625dd6c","session":"56.1.1","stderr":"Failed to create container: containerizer: wait for container: error: containerizer: initializing the container: system: mount sysfs on /sys: operation not permitted","stdout":"","took":"135.248818ms"}}
{"timestamp":"1449564439.654448271","source":"garden-linux","message":"garden-linux.container.start.wshd-start-failed","log_level":2,"data":{"error":"exit status 2","handle":"c3ae6bcd-5825-4510-81e2-5e13089c868d-5181b8b2-aa86-4a3a-8e8d-54ccfc1f6fc1-0835adbc-f0ce-4d89-4154-cba0d625dd6c","session":"56.1"}}
{"timestamp":"1449564439.654495716","source":"garden-linux","message":"garden-linux.pool.release.releasing","log_level":1,"data":{"id":"7f24gt0ouaj","session":"8.18"}}
{"timestamp":"1449564439.841723919","source":"garden-linux","message":"garden-linux.pool.release.subnet-pool.release.changing-allocated-subnets","log_level":1,"data":{"allocated-subnets":null,"id":"7f24gt0ouaj","session":"8.18.2.1"}}
{"timestamp":"1449564439.841737270","source":"garden-linux","message":"garden-linux.pool.release.released","log_level":1,"data":{"id":"7f24gt0ouaj","session":"8.18"}}
{"timestamp":"1449564439.841777563","source":"garden-linux","message":"garden-linux.garden-server.create.failed","log_level":2,"data":{"error":"container: start: exit status 2","request":{"Handle":"c3ae6bcd-5825-4510-81e2-5e13089c868d-5181b8b2-aa86-4a3a-8e8d-54ccfc1f6fc1-0835adbc-f0ce-4d89-4154-cba0d625dd6c","GraceTime":0,"RootFSPath":"docker://docker.libnux.org/goenv#latest","BindMounts":null,"Network":"","Privileged":false,"Limits":{"bandwidth_limits":{},"cpu_limits":{"limit_in_shares":10},"disk_limits":{"inode_hard":200000,"byte_hard":134217728},"memory_limits":{"limit_in_bytes":67108864}}},"session":"10.529"}}

Releases I used in my deployment
cf: 225
diego: 0.1441.0
etcd: 18
garden-linux: 0.327.0

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/109699090.

[libnux]

I found similar issue: #53. However I checked that the aufs fs driver was already loaded in the vagrant host vm.

I use bosh-lite. The boshlite image version is cloudfoundry/bosh-lite (virtualbox, 9000.69.0), and stemcell version is bosh-warden-boshlite-ubuntu-trusty-go_agent v3126

[goonzoid]

@libnux We're looking into this. What docker image were you using? If it's not public or shareable, is there something close that we can use to try to reproduce? Same base image perhaps?

EDIT: No worries, just noticed the rootfs path is in the logs!
EDIT 2: Looks like the docker registry at docker.libnux.org isn't available for us. If you can provide a docker image for us that's as similar as possible to the one you're seeing the problem with, we can try to recreate.

[libnux]

@goonzoid, please use the public libnux/goserver image to recreate. It's a very simple go http server based on busybox image.

cf push dockerapp -o docker-bj1.mi-ae.com.cn/goserver

Thanks!

[goonzoid]

@libnux I can run that image with no problems on the versions you describe. I'm wondering if you are definitely using the 9000.69.0 bosh-lite vagrant box? There was an issue with older versions of bosh-lite where /sys wasn't mounted inside the top level containers, which lead to problems similar to the one you're having.

I know vagrant sometimes has some surprising behaviour around box versions and when it updates. I'd recommend making sure that you really are running the most recent bosh-lite box before trying this again. I think vagrant box list will show you what boxes you have locally. Probably best to delete the older ones.

[libnux]

Cool! I delete the old version box, and did a fresh new deployment。 Everything worked.

Thanks a lot！ @goonzoid