K8TRE - base infrastructure options: Storage classes

[manics]

K8TRE will assume certain storage requirements can be fulfilled. Where possible these should be encoded in terms of standard Kubernetes abstractions (e.g. supported Access Modes) so that they can be implemented on any K8S cluster. Recommended implementations for the initial example platforms should also be defined.

[manics]

AWS:

• The EBS CSI controller provisions block storage volumes. EBS volumes can bee backed by different storage hardware with a price/performance tradeoff. They are single availability zone. In general a volume can only be mounted once though it may be possible to mount a volume to multiple pods/hosts.
• The EFS CSI controller is an NFS implementation, can be shared across availability zones, and mounted to multiple pods/hosts. It is more expensive than EBS storage.

[vvcb]

The equivalents for Azure are listed here - https://learn.microsoft.com/en-us/azure/aks/concepts-storage#storage-classes. The choices are disk (SSD), file share (Azure File Share and EFS are equivalent) and blob storage.

For most K8TRE use cases, we would expect storage to be mounted into the container using one of the CSI drivers.

The PVCs that any application running as part of K8TRE should request from a finite number of storage classes that we define and not simply from the default storage class. For instance, in the LSC-SDE implementation, JupyterHub uses an environment-specific storage class based on Azure File Share.

Below is an example of how we may do this in K8TRE. The application should simply see the mounted PV as a directory with identical filesystem semantics across all platforms.

Azure File Share

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: k8tre-fileshare
provisioner: file.csi.azure.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - dir_mode=0640
  - file_mode=0640
  - uid=0
  - gid=0
  - mfsymlinks
  - cache=strict # https://linux.die.net/man/8/mount.cifs
  - nosharesock
parameters:
  skuName: Standard_LRS

AWS EFS and FSx

Something like the below should work.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: k8tre-fileshare
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-xxxxxxxx  # Replace with actual EFS filesystem ID

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: k8tre-fileshare
provisioner: fsx.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
  subnetId: subnet-xxxx  # Required
  securityGroupIds: sg-xxxx  # Required
  deploymentType: SINGLE_AZ_2
  storageType: SSD

MicroK8s

For on-prem, the default hostPath storage option does not support ReadWriteMany. This will require NFS server to be setup. (I will test this as part of my setup this week). Below is what I got from both Claude and Gemma (running locally on my laptop).

Step 1: Install NFS Server

First, install NFS server software on the host:

sudo apt-get update
sudo apt-get install nfs-kernel-server

Step 2: Create an NFS Export Directory

Create a directory that will be exported via NFS:

sudo mkdir -p /srv/nfs/k8s-share
sudo chown nobody:nogroup /srv/nfs/k8s-share
sudo chmod 777 /srv/nfs/k8s-share # Adjust permissions as necessary

Step 3: Configure NFS Export

Edit the /etc/exports file to export the directory. Replace * with
the IP address or CIDR block of the client nodes if you want to restrict
access:

sudo nano /etc/exports

Add the following line:

/srv/nfs/k8s-share *(rw,sync,no_subtree_check)

Save and exit the editor.

Step 4: Restart NFS Server

Restart the NFS server to apply the changes:

sudo exportfs -a
sudo systemctl restart nfs-kernel-server

Optional:

Now install the NFS client on all your microk8s nodes:

sudo apt install nfs-common -y

Step 5: Create StorageClass in MicroK8s

Now, create a StorageClass that uses the NFS export. Here is an example
YAML file for the StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: k8tre-fileshare
provisioner: microk8s.io/nfs
parameters:
  server: <your-server-ip> # Need to sense check if this is required.
  path: /srv/nfs/kubedata
reclaimPolicy: Delete
volumeBindingMode: Immediate

Alternative to above would be to use something like Ceph which I have no experience with.

ToDo:

• Identify initial storage classes required - as a minimum they should support the following access modes - ReadWriteOnce, ReadWriteMany and ReadOnlyMany. K8S also has a ReadWriteOncePod option which I haven't needed yet.
• Decide on mount options for storage classes that use file shares and disks. This is important as it affects both performance as well as the ability of some tools to work (eg. git ). For instance, at LSC-SDE we use the following mount options for JupyterHub.

mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=1000
  - gid=100
  - mfsymlinks # Enable support for Minshall+French symlinks 
  - cache=strict # https://linux.die.net/man/8/mount.cifs
  - actimeo=30 # reduce latency for metadata-heavy workload
  - nosharesock # reduce probability of reconnect race

• Decide how K8TRE will implement this across diverse environments. I am veering towards ArgoCD's ApplicationSet generators with ArgoCD automatically deciding which storage classes (and other cluster-specific components) to deploy to which cluster using labels. This will also align with another ArgoCD recommended best practice - to not run user workloads on the same cluster as the ArgoCD management cluster. This pattern also gets us a little closer to the scenario where organisations wish to deploy a cluster per project.

[manics]

It's also possible to run NFS using the NFS Ganesha Helm chart, but it requires the nfs-common package to be installed on the host:
https://github.com/manics/jupyterhub-not-a-tre/blob/main/nfs-ganesha.yaml

[craddm]

We are using Longhorn in FRIDGE. It should work well across different platforms. It uses NFS to do ReadWriteMany storage classes.