Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Quobyte CSI

Quobyte CSI is the implementation of Container Storage Interface (CSI). Quobyte CSI enables easy integration of Quobyte Storage into Kubernetes. Current Quobyte CSI plugin supports the following functionality

  • Dynamic Volume Create
  • Volume Delete
  • Pre-provisioned volumes (Delete policy does not apply to these volumes)
  • Volume Expansion (Only dynamically provisioned volumes can be expanded)
    • Quobyte supports volumes with unlimited size, expanding an unlimited sized volume restricts the volume size to expanded size.
  • Volume snapshots

Select Quobyte CSI Driver Release

  1. Choose a Quobyte CSI release from available releases

  2. Follow the instructions specific to that release

release_instructions_image

Index

Requirements

  • Requires git on k8s master node
  • Requires at least Kubernetes v1.17
  • Quobyte installation with reachable registry and api services from the Kubernetes nodes and pods
  • Quobyte client with mount path as <values.clientMountPoint>/mounts. Please see Deploy Quobyte clients for Quobyte client installation instructions.
    • To use Quobyte access keys, the Quobyte client (requires Quobyte version 3.0 or above) should be deployed with --enable-access-contexts. Additionally, the metadata cache (global policy) should be disabled.
  • If you have load balancer for Quobyte API, the load balancer must be configured with sticky sessions.
  • Requires additional setup to use volume snapshots

Deploy Quobyte CSI Driver

Note: Quobyte CSI driver automatically deletes all the application pods with stale Quobyte CSI volumes and leaves the new pod creation to kubernetes. To reschedule a new pod automatically by k8s, applications should be deployed with Deployment/ReplicaSet/StatefulSets but not as a plain Pod.

  1. Set your chosen release version

    # For example, to install Quobyte CSI release v1.0.5,
    # please set RELEASE_TAG="v1.0.5"
    RELEASE_TAG="<YOUR_CHOSEN_RELEASE>"
  2. Clone the quobyte CSI repository from github on k8s master node

    git clone https://github.com/quobyte/quobyte-csi.git && cd quobyte-csi \
     && git checkout tags/$RELEASE_TAG
  3. Helm is required to deploy the Quobyte CSI driver. Please install Helm on the k8s master node.

    curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
    && chmod 700 get_helm.sh && ./get_helm.sh
  4. Edit Quobyte CSI driver configuration (./quobyte-csi-driver/values.yaml) and configure CSI driver with Quobyte API, other required information.

  5. (optional) generate driver deployment .yaml and verify the configuration.

    helm template ./quobyte-csi-driver --debug > csi-driver.yaml
  6. Deploy the Quobtye CSI driver (deploys driver with configuration from step 3)

    # Depolys helm chart with name "quobyte-csi".
    # Please change quobyte-csi as required
    helm install quobyte-csi ./quobyte-csi-driver
  7. Verify the status of Quobyte CSI driver pods

    Deploying Quobyte CSI driver should create a CSIDriver object with your csiProvisionerName (this may take few seconds)

    CSI_PROVISIONER="<YOUR-csiProvisionerName>"
    kubectl get CSIDriver | grep ^${CSI_PROVISIONER}

    The Quobyte CSI plugin is ready for use, if you see quobyte-csi-controller-x pod running on any one node and quobyte-csi-node-xxxxx running on every node of the Kubernetes cluster.

    CSI_PROVISIONER=$(echo $CSI_PROVISIONER | tr "." "-")
    kubectl -n kube-system get po -owide | grep ^quobyte-csi-.*-${CSI_PROVISIONER}
  8. Make sure your CSI driver is running against the expected Quobyte API endpoint

    kubectl -n kube-system exec -it \
    "$(kubectl get po -n kube-system | grep -m 1 ^quobyte-csi-node-$CSI_PROVISIONER \
    |  cut -f 1 -d' ')" -c quobyte-csi-plugin -- env | grep QUOBYTE_API_URL

    The above command should print your Quobyte API endpoint. After that, uninstall Quobyte CSI driver and install again.

Use Quobyte volumes in Kubernetes

Note: This section uses example/ deployment files for demonstration. These should be modified with your deployment configurations such as namespace, quobyte registry, Quobyte API user credentials etc.

We use quobyte namespace for the examples. Create the namespace

kubectl create ns quobyte

Quobyte requires a secret to authenticate volume create and delete requests. Create this secret with your Quobyte API login credentials (Kubernetes requires base64 encoding for secret data which can be obtained with the command echo -n "value" | base64). Please encode your user name, password (and optionally access key information) in base64 and update example/csi-secret.yaml. If provided, access key ensures only authorized user can access the tenant and volumes (users must be restricted to their own namespace in k8s cluster).

kubectl create -f example/csi-secret.yaml

Create a storage class with the provisioner set to csi.quobyte.com along with other configuration parameters. You could create multiple storage classes by varying parameters such as quobyteTenant, quobyteConfig etc.

kubectl create -f example/StorageClass.yaml

To run the Nginx demo pods,

  1. Host nodes must have nginx user (UID: 101) and group (GID: 101). Please create nginx user and group on every node.

    sudo groupadd -g 101 nginx; sudo useradd -u 101 -g 101 nginx
  2. nginx user must have at least read and execute permissions on the volume

Dynamic volume provisioning

Creating a PVC referencing the storage class created in the previous step would provision dynamic volume. The secret csi.storage.k8s.io/provisioner-secret-name from the namespace csi.storage.k8s.io/provisioner-secret-namespace in the referenced StorageClass will be used to authenticate volume creation and deletion.

  1. Create PVC to trigger dynamic provisioning

    kubectl create -f example/pvc-dynamic-provision.yaml
  2. Mount the PVC in a pod as shown in the following example

    kubectl create -f example/nginx-demo-pod-with-dynamic-vol.yaml
  3. Wait for the pod to be in running state

    kubectl get po -w | grep 'nginx-dynamic-vol'
  4. Once the pod is running, copy the index file to the deployed nginx pod

    kubectl cp example/index.html nginx-dynamic-vol:/usr/share/nginx/html/
  5. Access the home page served by nginx pod from the command line

    curl http://$(kubectl get pods nginx-dynamic-vol -o yaml | grep ' podIP:' | awk '{print $2}'):80

Above command should retrieve the Quobyte CSI welcome page (in raw html format).

Use existing volumes

Quobyte CSI requires the volume UUID to be passed on to the PV as VolumeHandle

  • Quobyte-csi supports both volume name and UUID
    • To use Volume Name VolumeHandle should be of the format <Tenant_Name/UUID>|<Volume_Name> and nodePublishSecretRef with Quobyte API login credentials should be specified as shown in the example PV example/pv-existing-vol.yaml
    • To use Volume UUID VolumeHandle can be |<Volume_UUID>.

In order to use the pre-provisioned test volume belonging to the tenant My Tenant, user needs to create a PV with volumeHandle: My Tenant|test as shown in the example PV.

  1. Edit example/pv-existing-vol.yaml and point it to the the pre-provisioned volume in Quobyte storage through volumeHandle. Create the PV with pre-provisioned volume.

    kubectl create -f example/pv-existing-vol.yaml
  2. Create a PVC that matches the storage requirements with the above PV (make sure both PV and PVC refer to the same storage class). The created PVC will automatically binds to the PV.

    kubectl create -f example/pvc-existing-vol.yaml
  3. Create a pod referring the PVC as shown in the below example

    kubectl create -f example/nginx-demo-pod-with-existing-vol.yaml
  4. Wait for the pod to be in running state

    kubectl get po -w | grep 'nginx-existing-vol'
  5. Once the pod is running, copy the index file to the deployed nginx pod

    kubectl cp example/index.html nginx-existing-vol:/usr/share/nginx/html/
  6. Access the home page served by nginx pod from the command line

    curl http://$(kubectl get pods nginx-existing-vol -o yaml | grep ' podIP:' | awk '{print $2}'):80

    The above command should retrieve the Quobyte CSI welcome page (in raw html format).

Volume snapshots

Snapshot Requirements

  1. Quobyte CSI Driver is deployed with enableSnapshots: true

  2. Snapshotter setup

Dynamic Snapshots

  1. Provision a PVC for a Quobyte volume by following the instructions

  2. Populate backing volume with nginx index file

    VOLUME="<Quobyte-Volume>" # volume for which snapshot will be taken
    wget https://raw.githubusercontent.com/quobyte/quobyte-csi/master/example/index.html -P <values.clientMountPoint>/mounts/$VOLUME
  3. Create volume snapshot secrtes

    Our examples use same secret in all the places wherever secret is required. Please create and configure secrets as per your requirements.

    kubectl create -f example/csi-secret.yaml
  4. Create volume snapshot class

    kubectl create -f example/volume-snapshot-class.yaml
  5. Create dynamic volume snapshot

    kubectl create -f example/volume-snapshot-dynamic-provision.yaml

    The above command should create required volumesnapshotcontent object dynamically

  6. (optional) verify created volumesnapshot and volumesnapshotcontent objects

    kubectl get volumesnapshot
    kubectl get volumesnapshotcontent
  7. Restore snapshot and create PVC

    kubectl create -f example/restore-snapshot-pvc-dynamic-provision.yaml

    This should create a PVC and a PV for the restored snapshot

  8. Create pod with restored snapshot

    kubectl create -f example/nginx-demo-pod-with-dynamic-snapshot-vol.yaml

Pre-provisioned Snapshots

  1. Create volume snapshot class

    kubectl create -f example/volume-snapshot-class.yaml
  2. Create volume snapshot secrtes

    Our examples use same secret in all the places wherever secret is required. Please create and configure secrets as per your requirements.

    kubectl create -f example/csi-secret.yaml
  3. Create VolumeSnapshotContent object for pre-provisioned volume with required configuration

    kubectl create -f example/volume-snapshot-content-pre-provisioned.yaml
  4. Create VolumeSnapshot object by adjusting the example snapshot object

    name and namespace must match volumeSnapshotRef details from the step 2

    kubectl create -f example/volume-snapshot-pre-provisioned.yaml
  5. (optional) verify created volumesnapshot and volumesnapshotcontent objects

    kubectl get volumesnapshot
    kubectl get volumesnapshotcontent
  6. Restore snapshot

    kubectl create -f example/restore-snapshot-pvc-pre-provisioned.yaml
  7. Create pod with restored snapshot

    kubectl create -f example/nginx-demo-pod-with-pre-provisioned-snapshot-vol.yaml

Uninstall Quobyte CSI

  1. Delete Quobyte CSI containers and corresponding RBAC

    List available helm charts

    helm list

    Delete intended chart

    helm delete <Quobyte-CSI-chart-name>

Snapshotter Setup

Install Snapshotter

The below setup is required once per k8s cluster

  # https://github.com/kubernetes-csi/external-snapshotter/
  kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v3.0.3/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml;
  kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v3.0.3/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml;
  kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v3.0.3/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml;
  kubectl create -f quobyte-csi-driver/k8s-snapshot-controller.yaml

Remove Snapshotter

  kubectl delete -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v3.0.3/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml;
  kubectl delete -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v3.0.3/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml;
  kubectl delete -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v3.0.3/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml;
  kubectl create -f quobyte-csi-driver/k8s-snapshot-controller.yaml