setting quota limit for projid 2: function not implemented

[teddyking]

NB: Are you currently experiencing this issue? Please reach out to us on the #garden slack channel before attempting any of the temporary resolutions or before destroying affected the cell(s). We haven't been able to reproduce this issue ourselves and would really appreciate an opportunity to do some live debugging! Thanks!

---

Environment

• IaaS: vSphere
• Stemcell: 3468.21
• Kernel: 4.4.0-111-generic

Symptom

Failed to create container: running image plugin create: making image: creating image: applying disk limits: apply disk limit: <nil>:  setting quota to /var/vcap/data/grootfs/store/unprivileged/images/b1639404-2935-11e8-b467-0ed5f89f718b: setting quota limit for projid 2: function not implemented: exit status 1

Cause

TBC

Resolution

TBC

Temporary workaround

This issue occurs when garden is configured to use GrootFS for image management.
It is possible to fall back to the previous image managemtn tool (garden-shed) by setting the following BOSH property:

garden.deprecated_use_garden_shed = true

NB It is recommended to force a recreate of VMs running garden (typically the diego cell VMs) when switching between GrootFS and garden-shed.

Additional info

guardian         [ERROR] 03/15 16:50:56.27 577.2.1    guardian.create.volume-creator.image-plugin-create.grootfs.create.groot-creating.making-image.overlayxfs-creating-image.applying-quotas.run-tardis.tardis.set-quota.setting-quota-failed
                                                      Error: setting quota limit for projid 2: function not implemented

guardian         [ERROR] 03/15 16:50:56.27 577.2.1    guardian.create.volume-creator.image-plugin-create.grootfs.create.groot-creating.making-image.overlayxfs-creating-image.applying-quotas.run-tardis.tardis-failed
                                                      Error: exit status 1

guardian         [ERROR] 03/15 16:50:56.27 577.2.1    guardian.create.volume-creator.image-plugin-create.grootfs.create.groot-creating.making-image.creating-image-failed
                                                      Error: applying disk limits: apply disk limit: <nil>:  setting quota to /var/vcap/data/grootfs/store/unprivileged/images/b1639404-2935-11e8-b467-0ed5f89f718b: setting quota limit for projid 2: function notimplemented: exit status 1

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/156048188

The labels on this github issue will be updated when the story is started.

[Callisto13]

Update

(Potential) Cause

This error (funtion not implemented) can present on three occasions:

1. When the kernel has been compiled without the CONFIG_QUOTA option.

• we do not believe this is likely otherwise we would be seeing this consistently on every VM using XFS quotas.

2. When the filesystem is mounted without the pquota (or prjquota) option.

• The mount tables of affected VMs show that option is present on all XFS mountpoints.
• note: when remounting an already mounted XFS filesystem, XFS will ignore any option to turn quotas on or off.

3. When no directory path on which to impose quotas is passed to xfs_quota.

• This is unlikely since the tool which wraps our quota syscalls (Tardis) will blow up if that path is not provided.

It is possible that multiple starts of the garden ctl script (which may happen if the initial start takes too long to complete) could have caused multiple XFS mountpoints to be layered over each other, thus obscuring a bad set of mount options. Groot will error if an attempt is made to re-init an already existing store, but may not if multiple inits run concurrently.
This may be unlikely since GrootFS hardcodes the pquota option.

(Temporary) Resolution

Since we do not have free access to any environment in which this problem presents, we are going to act defensively and ensure that BOSH fails if the GrootFS store has not been created properly. Right now, the problem is not visible until the first app push.

This resolution is in release 1.13.1 onwards.
We would still advise to recreate VMs on any upgrade of Garden-Runc-Release.

The investigation is still ongoing.