grootfs failing to calculate base size for volume due to missing metadata file

[sunjayBhatia]

Description

grootfs started failing in our CI environment on create due to the following error:

running image plugin create: making image: creating image: generating lowerdir paths failed: calculating base volume size for volume f5c75f24dcd655ef23ddd5a6cadd8b14113768897ebb1098dda290207dafec4c: open /var/vcap/data/grootfs/store/unprivileged/meta/volume-f5c75f24dcd655ef23ddd5a6cadd8b14113768897ebb1098dda290207dafec4c: no such file or directory\n: exit status 1

Note: the volume id above corresponds to the cflinuxfs2 rootfs image volume.

This was a repeated error that caused a Diego cell in our CF environment to be unhealthy for ~4 days as it occurred on all container creations during that time (containers were being created by the Diego->Garden healthcheck). It was resolved by a redeploy of the cell (the cell was not recreated).

At the moment we do not know what the root cause is, we have tried to look through the logs but can't discern any behavior that stands out. It is worth noting that around the affected time period, the above error is the first to occur, there are no other partial create/destroy errors that we can see in the guardian logs.

Environment

• garden-runc-release version: 1.17.2
• IaaS: GCP
• Stemcell version: ubuntu-xenial 170.16
• Kernel version: 4.15.0-43-generic

Steps to reproduce

N/A

Links

• Slack thread: https://cloudfoundry.slack.com/archives/C033RE5D6/p1547566683143600
• Diego team investigation story: https://www.pivotaltracker.com/story/show/163212244

Logs

• Diego cell unhealthiness: https://kibana.logit.io/goto/ae64ac72db768a6a3c70192f13062ac1
• Logs referencing volume with missing metadata file: https://kibana.logit.io/goto/d5828379057763c5c3200a8e4b01195d
• Logs referencing volume with missing metadata file with errors: https://kibana.logit.io/goto/a032ec0f127ba5ad83795cef5431e04d
• Logs from guardian surrounding the unhealthy period: https://kibana.logit.io/goto/b85b408719f8c81542605051cc9b94b8

We can give the Garden team access to the provided logs as needed and more if needed as well

Cause

N/A

Resolution

N/A

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/163232754

The labels on this github issue will be updated when the story is started.

[Callisto13]

Resolution

Fixed in 1.18.1