HostPath does not enforce quotas

[jshin47]

OpenEBS LocalPV HostPath do not enforce quotas
This is a known limitation of OpenEBS LocalPV HostPath, but it's one that seriously limits its usefulness in production.

I'd like for OpenEBS LocalPV HostPath to collect disk usage stats and render pods unscheduleable when they exceed quotas
I know this is not as straight forward as it sounds, but I'd like to make OpenEBS HostPath render pods unschedulable when quotas are exceeded. For existing pods, I'd like to kill them if they are exceeding quotas. For pods to be scheduled, I'd like for this quota to be enforced in advance.

Anything else you would like to add:
I know this is a roadmap feature.

[kmova]

Started experimenting with the feature of creating a sparse file within the hostpath to enforce quota. (inline with what Rawfile Local Pv does).

The PV spec will be changed to:

    local:
      fsType: ""
      path: /var/openebs/local/pvc-4ae3d213-0a93-4b77-9851-d5af29fc07f1/disk.img

The above results in Kubelet rejecting to mount the device with the following error:

MountVolume.NewMounter initialization failed for volume "pvc-4ae3d213-0a93-4b77-9851-d5af29fc07f1" : only directory and block device are supported

Need to figure out if I can easily convert the sparse file into a block device.

The current code is located at: https://github.com/kmova/dynamic-localpv-provisioner/tree/enforce-quota

[jshin47]

Thanks! I'll see if I can help in any way. I doubt it, but I'd like to learn how to set up a dev environment for this anyway because we may be using it heavily.

One thing that comes to mind is to use losetup -fP and then use that loopback device as your block device

[kmova]

Another related question with hostpath, which filesystem do you plan to use - xfs, ext4 or anything else.

Specifically, would it be ok to use XFS - as there is a possibility of using set quota on top of XFS.

[ankur-gupta-guavus]

I prefer using XFS only

[jshin47]

I use XFS only as well.

[kmova]

This feature also needs to account for

• volume resize

This feature needs a good implementation of generating unique xfs project ids per node, that are also unique across clusters in cases the PVs are migrated/merged onto same node.

[almas33]

Hi, I have submitted an application and excited to work on this. 😄

[kmova]

Implementation notes:

Prerequisites:

• The base path used by the provisioner should be mounted with XFS project quotas enabled.
  Refer: https://fabianlee.org/2020/01/13/linux-using-xfs-project-quotas-to-limit-capacity-within-a-subdirectory/

  Example:

  sudo mkfs.xfs /dev/sdb
  sudo mkdir -p /mnt/xfs_hostpath
  sudo mount -o pquota /dev/sdb /mnt/xfs_hostpath
  

• Approach 1:

  • Provision the volume. Example : /mnt/xfs_hostpath/pvc-uuid-xyz
  • Launch another job that sets the project quota on the volume directory.
    Example:

    sudo xfs_quota -x -c 'project -s -p /mnt/xfs_hostpath/vol5m 100' /mnt/xfs_hostpath
    sudo xfs_quota -x -c 'limit -p bsoft=5m bhard=5m 100' /mnt/xfs_hostpath
    

• Approach 2:

  • Provision the volume. Example : /mnt/xfs_hostpath/pvc-uuid-xyz
  • Run a daemon set that watches for new PVCs created on the node that sets the project quota on the volume directory.
    Example:

    sudo xfs_quota -x -c 'project -s -p /mnt/xfs_hostpath/vol5m 100' /mnt/xfs_hostpath
    sudo xfs_quota -x -c 'limit -p bsoft=5m bhard=5m 100' /mnt/xfs_hostpath
    

In both of the above cases, the solution needs to take care of finding a unique project id to be associated with a given volume.

Also for inspiration on how to set XFS project quota you can check the following:
https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/blob/master/pkg/volume/quota.go

[almas33]

This feature also needs to account for

* volume resize

@kmova
currently quota pod is running just after helper pod by taking parameters from storage class
for this feature should I also add an standalone quota pod which by taking name of pv can resize the limits of quota if user wants to?

[Ab-hishek]

@jshin47 Hostpath now supports enforcing quota for xfs fs in OpenEBS 3.0 release. Please check the latest release and if everything seems good to you can you close this issue?

[jshin47]

@Ab-hishek Thank you so much! We will test in the next week or so and confirm it works. Thanks again, this will be so helpful!

[Ab-hishek]

This feature seems to be working fine. Many users have already started using this feature and it works fine. Therefore closing the issue.

[Icedroid]

This feature also needs to account for

* volume resize

@kmova currently quota pod is running just after helper pod by taking parameters from storage class for this feature should I also add an standalone quota pod which by taking name of pv can resize the limits of quota if user wants to?

Openebs hostpath now support volume resize?

[Ab-hishek]

@Icedroid No, its not exactly resize. The quotas(supported for xfs file systems only to be precise) limits the amount of storage hostpath volumes can consume instead of using the entire underlying storage.
References:

1. feat(hostpath):enforce xfs quota on the hostpath #78
2. https://github.com/openebs/dynamic-localpv-provisioner/tree/develop/docs/tutorials/hostpath/xfs_quota

[aleksasiriski]

Could support be added for BTRFS as well? BTRFS support quotas via subvolumes and has easy volume snapshots.