Add overlay2.inode storage opt (xfs)

[cirocosta]

Nowadays it's possible to set project quotas if the underlying filesystem is XFS and uses overlay2 as the graph driver (see https://github.com/moby/moby/blob/master/daemon/graphdriver/overlay2/overlay.go) but it only allows setting a limit on disk blocks.

That's very good but we could extend the functionality to also support inodes as it's just an extra field defined in fs_disk_quota:

typedef struct fs_disk_quota {
// ...
	__u64		d_blk_hardlimit;/* absolute limit on disk blks */
	__u64		d_blk_softlimit;/* preferred limit on disk blks */
	__u64		d_ino_hardlimit;/* maximum # allocated inodes */
	__u64		d_ino_softlimit;/* preferred inode limit */
// ...
} fs_disk_quota_t;

(from https://github.com/torvalds/linux/blob/42ff72cf27027fa28dd79acabe01d9196f1480a7/include/uapi/linux/dqblk_xfs.h#L57-L60)

My proposal is to update Quota to include inodes:

moby/daemon/graphdriver/quota/projectquota.go

Lines 64 to 66 in 184cea5

	type Quota struct {
	Size uint64
	}

In overlay.go we could check for the option overlay2.inodes

moby/daemon/graphdriver/overlay2/overlay.go

Lines 233 to 240 in 184cea5

	case "overlay2.size":
	size, err := units.RAMInBytes(val)
	if err != nil {
	return nil, err
	}
	o.quota.Size = uint64(size)
	default:
	return nil, fmt.Errorf("overlay2: unknown option %s", key)

and then set Quota.Inodes properly.

With that it's a matter of setting the values in the struct:

moby/daemon/graphdriver/quota/projectquota.go

Lines 180 to 181 in 184cea5

	d.d_blk_hardlimit = C.__u64(quota.Size / 512)
	d.d_blk_softlimit = d.d_blk_hardlimit

(naturally that should come with some bound checking, etc).

I'd be willing to implement the feature if desired.

Wdyt?

Thx!

---

Update: we also need to update the bitmask:

moby/daemon/graphdriver/quota/projectquota.go

Line 179 in 184cea5

d.d_fieldmask = C.FS_DQ_BHARD | C.FS_DQ_BSOFT

so that it includes

#define FS_DQ_ISOFT	(1<<0)
#define FS_DQ_IHARD	(1<<1)

[AkihiroSuda]

What would be usecases?

[cirocosta]

Hey @AkihiroSuda , my use case is placing a hard limit on the number of inodes that a user can create in its container such that it can't affect too much the underlying fs (as that counter is shared with all containers as I see - let me know if I got this wrong). When a user creates a container with a huge amount of files (even though they're not open) it takes a lot of time to rm the container, which is undesirable for us.

It's already possible to limit the number of open file descriptors via ulimits (RLIMIT_NOFILE) but that's only for files that are kept open. xfs is capable of dynamically allocating inodes (such that you don't necessarily hit a 100% INode usage easily) but that still means that a container would be able to own too much resources for itself and end up impacting the host.

[markzhang0928]

hey @cirocosta , I followed your instructions to place a hard limit on the number of inodes for each container which backingfs is xfs. And I can see the limit of Inodes had already been set. BUT I found that the sum of free inodes is wrong when i use the command df -i . I guess it may be the problem of overlayfs, but I still can't fix it. Do you have any suggestions?

root@ubuntujoeypc:/home/joeypc/go/src/github.com/docker/docker/bundles# df -iT

Filesystem                        Type       Inodes     IUsed    IFree IUse% Mounted on
udev                              devtmpfs  1400011       495  1399516    1% /dev
tmpfs                             tmpfs     1404809       790  1404019    1% /run
/dev/mapper/ubuntujoeypc--vg-root ext4     32555008    272103 32282905    1% /
tmpfs                             tmpfs     1404809         2  1404807    1% /dev/shm
tmpfs                             tmpfs     1404809         3  1404806    1% /run/lock
tmpfs                             tmpfs     1404809        16  1404793    1% /sys/fs/cgroup
/dev/sdb1                         ext2       124928       298   124630    1% /boot
tmpfs                             tmpfs     1404809        12  1404797    1% /run/lxcfs/controllers
tmpfs                             tmpfs     1404809         4  1404805    1% /run/user/1000
/home/joeypc/.Private             ecryptfs 32555008    272103 32282905    1% /home/joeypc
/dev/sda                          xfs      20971520     74411 20897109    1% /data
overlay                           overlay    102400 -20794702 20897102     - /data/docker/overlay2/a0cd292a180f674c5eca6f40c150ad154ba5bfeb443589afa69bcac5b108f729/merged
shm                               tmpfs     1404809         1  1404808    1% 
/data/docker/containers/2a03e7df1773cb965c7bb69caabfe56a958b86eeb6ea1ae9634d33584dd1f860/shm

---

some research updated. https://my.oschina.net/markz0928/blog/3045719