Failed to start container with empty /etc/

[keymon]

Description

When using a container with an empty /etc/ directory, the application crashes starting with a cryptic message:

[CELL/0] ERR Cell 6bdffa08-608d-4740-af08-3b289d433ec5 failed to create container for instance d52fd44a-eabb-4bdd-4360-6185: runc run: exit status 1: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/vcap/packages/healthcheck\\\" to rootfs \\\"/var/vcap/data/grootfs/store/unprivileged/images/d52fd44a-eabb-4bdd-4360-6185/rootfs\\\" at \\\"/var/vcap/data/grootfs/store/unprivileged/images/d52fd44a-eabb-4bdd-4360-6185/rootfs/etc/cf-assets/healthcheck\\\" caused \\\"mkdir /var/vcap/data/grootfs/store/unprivileged/images/d52fd44a-eabb-4bdd-4360-6185/rootfs/etc/cf-assets: permission denied\\\"\""

Environment

• garden-runc-release version: 1.13.3
• IaaS: AWS
• Stemcell version: 3586.24
• Kernel version:

$ uname -a
Linux c763447a-1984-41d5-b2fd-586c47afe31b 4.4.0-128-generic #154~14.04.1-Ubuntu SMP Fri May 25 14:58:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce

• Create a container with an empty /etc/ directory (e.g. keymon/empty-etc)
• Start the container: cf push test-container -o keymon/empty-etc

Logs

$ cf push test-container -o keymon/empty-etc
Pushing app test-container to org admin / space billing as admin...
Getting app info...
Updating app with these attributes...
  name:                test-container
- docker image:        hector/empty-etc
+ docker image:        keymon/empty-etc
  disk quota:          1G
  health check type:   port
  instances:           1
  memory:              1G
  stack:               cflinuxfs2
  routes:
    test-container.hector.dev.cloudpipelineapps.digital

Updating app test-container...
Mapping routes...

Staging app and tracing logs...
   Cell cde8f71a-a756-4891-b130-8f6ac230eb59 successfully created container for instance 48f69ea7-aaba-4845-b2f5-90761daa7c46
   Staging...
   Staging process started ...
   Staging process finished
   Exit status 0
   Staging Complete
   Cell cde8f71a-a756-4891-b130-8f6ac230eb59 stopping instance 48f69ea7-aaba-4845-b2f5-90761daa7c46
   Cell cde8f71a-a756-4891-b130-8f6ac230eb59 destroying container for instance 48f69ea7-aaba-4845-b2f5-90761daa7c46
   Cell cde8f71a-a756-4891-b130-8f6ac230eb59 successfully destroyed container for instance 48f69ea7-aaba-4845-b2f5-90761daa7c46

Waiting for app to start...
Start unsuccessful

TIP: use 'cf logs test-container --recent' for more information
FAILED

$ cf logs test-container --recent
....
   2018-07-03T09:27:05.42+0100 [CELL/0] OUT Cell cde8f71a-a756-4891-b130-8f6ac230eb59 creating container for instance afd31154-83ac-425a-66d9-2df0
   2018-07-03T09:27:07.07+0100 [CELL/0] ERR Cell cde8f71a-a756-4891-b130-8f6ac230eb59 failed to create container for instance afd31154-83ac-425a-66d9-2df0: runc run: exit status 1: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/vcap/packages/healthcheck\\\" to rootfs \\\"/var/vcap/data/grootfs/store/unprivileged/images/afd31154-83ac-425a-66d9-2df0/rootfs\\\" at \\\"/var/vcap/data/grootfs/store/unprivileged/images/afd31154-83ac-425a-66d9-2df0/rootfs/etc/cf-assets/healthcheck\\\" caused \\\"mkdir /var/vcap/data/grootfs/store/unprivileged/images/afd31154-83ac-425a-66d9-2df0/rootfs/etc/cf-assets: permission denied\\\"\""
   2018-07-03T09:27:07.08+0100 [CELL/0] OUT Cell cde8f71a-a756-4891-b130-8f6ac230eb59 destroying container for instance afd31154-83ac-425a-66d9-2df0
   2018-07-03T09:27:07.08+0100 [CELL/0] OUT Cell cde8f71a-a756-4891-b130-8f6ac230eb59 successfully destroyed container for instance afd31154-83ac-425a-66d9-2df0
   2018-07-03T09:27:07.11+0100 [API/0] OUT Process has crashed with type: "web"
   2018-07-03T09:27:07.11+0100 [API/0] OUT App instance exited with guid 87132572-9f9c-42d3-afdf-258137144597 payload: {"instance"=>"afd31154-83ac-425a-66d9-2df0", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"failed to initialize container: runc run: exit status 1: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"/var/vcap/packages/healthcheck\\\\\\\" to rootfs \\\\\\\"/var/vcap/data/grootfs/store/unprivileged/images/afd31154-83ac-425a-66d9-2df0/rootfs\\\\\\\" at \\\\\\\"/var/vcap/data/grootfs/store/unprivileged/images/afd31154-83ac-425a-66d9-2df0/rootfs/etc/cf-assets/healthcheck\\\\\\\" caused \\\\\\\"mkdir /var/vcap/data/grootfs/store/unprivileged/images/afd31154-83ac-425a-66d9-2df0/rootfs/etc/cf-assets: permission denied\\\\\\\"\\\"\"\n", "crash_count"=>7, "crash_timestamp"=>1530606427101357593, "version"=>"a0f025d4-e7cf-4b18-8093-5196e1569518"}

Docker image (if relevant)

keymon/empty-etc

Cause

It might be an issue with the layered fs, where an empty directory does not allow to mount a fs, or gets created with the wrong permissions.

Resolution

If instead you drop any file into /etc/, it does not throw that error, but one complaining about the missing /etc/passwd (which is fine).

FROM cbaines/govuk-mini-environment-admin:release_3-test2
RUN touch /etc/foo

unable to find user root: no matching entries in passwd file

Then if we add a valid /etc/passwd the image works.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/158788438

The labels on this github issue will be updated when the story is started.

[julz]

Hi @keymon - I suspect the problem here is that diego expects /etc/ to exist in order to bind mount /etc/cf-assets to it, so it can put the lifecycle binaries in to the container. In theory we should create the /etc/ parent directory as needed, but it looks like that failed. In know that in the deprecated garden-shed driver there's a problem (due to a limitation in aufs) that means we lack permissions in the container to create new directories directly under /, are you by any chance using garden-shed rather than the new grootfs driver?

[julz]

Going to close this due to inactivity @keymon feel free to re-open if your problem is not solved