Skip to content

cluster02: Longhorn and various NFS providers

Jump to bottom
pagong edited this page Mar 1, 2019 · 3 revisions

cluster02: Longhorn and various NFS providers

After having been overwhelmed by Ceph's complexity, I've implemented this cluster in Fall of 2018, using Rancher's longhorn v0.3.0.

Since I was going to have 3-way replicated volumes, I used 1 virtual disk of 50GB on each of the 3 kvm guests. Setup of the storage orchestrator has been straight forward, except for the pitfalls with RancherOS, that was used as operating system for the K8s cluster nodes.

There is an issue with using the correct path for the flexvolume plugins directory on RancherOS. You really have to read carefully and understand this note. And follow all of the steps in that note precisely!

This small code snippet in the cluster.yaml file for RKE is absolutely essential:

kubelet: 
  extra_args: 
    feature-gates: "MountPropagation=true"
    volume-plugin-dir: "/opt/rke/var/lib/kubelet/volumeplugins"

Exactly the same path has to be used for the FLEXVOLUME_DIR in the YAML file for the longhorn deployment:

- name: FLEXVOLUME_DIR
  value: "/opt/rke/var/lib/kubelet/volumeplugins/"

After having mastered the longhorn storage provisioner, I have tried several NFS server implementations, with various degrees of frustration and success.

Kubernetes examples: nfs-server-rc

First I tried the example code from the Kubernetes documentation nfs-server-rc. This attempt was not successful, as the provisioner pod relies on the NFS server code in the underlying node OS. I did not get this working on RancherOS.

Kubernetes incubator's external-storage: nfs-server-provisioner

Next I used the nfs-provisioner from the K8s incubator. The provisioner pod comes with it's own implementation of a NFS server: nfs-ganesha.

I used the helm chart version of the nfs-server-provisioner and got it working on RancherOS and Longhorn.

The problem with this implementation, is the usage of TCP and UDP ports in the docker binary. The metalLB load balancer is unable to deal with mixed TCP and UDP ports.

Another quirk is the order of steps that must be followed when creating a NFS share. First you have to create a PVC using the nfs-server storage class, before you can see the PV and share being exported.

That's why I recommend using the nfs-server-provisioner only for intra-cluster NFS shares.

Rook for NFS: rook-nfs

Finally I turned to Rook again. With release v0.9 of the Rook project, they have implemented support for several new storage providers, besides Ceph. One of them is a NFS storage provider, also based on nfs-ganesha.

This provisioner, called rook-nfs, is better than the nfs-server-provisioner, for three reasons:

  1. It is quite easy to have several instances of rook-nfs running. Each one needs it's own namespace and can serve multiple NFS exports from it's external IP.
  2. Since the docker binary is using only TCP ports, metalLB is able to assign an external IP to each NFS server's service.
  3. Since the underlying longhorn PVs of the NFS exports are directly exported by ganesha, there is no need to create dummy PVs and PVCs for the NFS clients.

That's why I recommend using the rook-nfs provisioners for inter-cluster (and intra-cluster) NFS shares.

Clone this wiki locally