Persistent Volumes for Kubernetes

Persistent volumes let pods create storage that persists beyond the lifetime of a pod. As a Photon project user, you can create persistent disks on Photon Platform and expose them as static persistent volumes in Kubernetes.

After you deploy a Kubernetes cluster, you can create dynamic persistent volumes entirely within Kubernetes without the intervention of an administrator. A tenant administrator, however, must provision a quota that includes persistent disks and storage resources.

For Kubernetes-related information about using persistent volumes, see Persistent Volumes.

Configuring the Master with Your Credentials

To create and use persistent volumes in a Kubernetes cluster on Photon Platform, you must first place your Photon Controller credentials in a configuration file on the Kubernetes master VM. Here's how:

Create a file named pc_login_info that contains, on the first line, your Photon Controller username and, on the second line, your Photon Controller password; example:

administrator@example.com
MySecret1!

Copy the file to the Kubernetes master VM by using the private key associated with the public key that was used to create the Kubernetes cluster; example:

scp -i kubernetes.privkey pc_login_info root@10.159.36.5:/etc/kubernetes/pc_login_info

Setting a Quota with Persisent Disks and Storage Capacity

As a tenant administrator, you need to set a quota that specifies, among other things, perisistent disks and storage capacity for your tenant. Here's an example:

photon tenant quota set <tenant-name> -l 'vm 100 COUNT, \
vm.cpu 100 COUNT, vm.memory 100 GB, ephemeral-disk.capacity 100 GB, \
ephemeral-disk 100 COUNT, persistent-disk.capacity 100 GB,  \
persistent-disk 100 COUNT, storage.LOCAL_VMFS 100 COUNT, \
storage.SHARED_VMFS 100 COUNT, storage.VSAN 100 COUNT, \
storage.NFS 100 COUNT'

Creating a Dynamically Provisioned Persistent Volume

Here's an example of how to create a dynamically provisioned persistent volume in Kubernetes:

kubectl apply -f persistent-volume-claim.yaml

Content of the persistent-volume-claim.yaml file:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: demo-claim
    annotations:
      volume.beta.kubernetes.io/storage-class: shared-vmfs
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
         storage: 2Gi

In this example, shared-vmfs denotes the storage class to be used for creating the persistent volume. Based on the default storage classes provided by Photon Controller, it can be one of the following: shared-vmfs, local-vmfs, vsan, nfs, default. If it is not specified, the persistent volume claim will be created with the default storage class shown below. When a persistent volume claim is created with the default storage class, the underlying persistent disk in photon controller can be created on any datastore.

kubectl get storageclass
NAME          KIND
default       StorageClass.v1.storage.k8s.io
local-vmfs    StorageClass.v1.storage.k8s.io
nfs           StorageClass.v1.storage.k8s.io
shared-vmfs   StorageClass.v1.storage.k8s.io
vsan          StorageClass.v1.storage.k8s.io

You can verify that the persistent volume claim named demo-claim was created:

kubectl get pvc
NAME         STATUS    VOLUME          CAPACITY   ACCESSMODES   AGE
demo-claim   Bound     pvc-b3ae54...   2Gi        RWO           18s

The claim has a 2Gi capacity, as specified. Since it is a dynamically created persistent volume, it also creates the corresponding persistent volume in Kubernetes and a corresponding persistent disk in Photon Controller, as you can see from the output of the following two commands:

kubectl get pv
NAME            CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                REASON    AGE
pvc-b3ae5...    2Gi        RWO           Delete          Bound     default/demo-claim             39m

photon disk list
ID                                    Name                             State
ddc133d9-b0e1-420b-b3d7-93e1284e9168  kubernetes-dynamic-pvc-b3ae5..   DETACHED

Creating a Pod That Uses a Persistent Volume

To use the dynamically created persistent volume in a pod, you can, for example, create an nginx pod with the persistent volume claim:

kubectl apply -f nginx-pod.yaml

Here's the content of nginx-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-demo
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
      hostPort: 80
    volumeMounts:
    - name: photon-storage
      mountPath: /data
  volumes:
  - name: photon-storage
    persistentVolumeClaim:
      claimName: demo-claim

You can get information about the pod.

kubectl get pod
NAME         READY     STATUS    RESTARTS   AGE
nginx-demo   1/1       Running   0          26s

The disk is attached and being used by the pod:

photon disk list
ID          Name                             State
ddc133d...  kubernetes-dynamic-pvc-b3ae5...  ATTACHED

Deleting the Persistent Volume

To delete the persistent volume, you first delete the pod, which will unmount the disk and detach it from the Kubernetes VM. Then you can delete the persistent volume claim, which in turn deletes the persistent disk from Photon Controller. Examples:

kubectl delete pod nginx-demo
kubectl delete pvc demo-claim

Creating a Statically Provisioned Persistent Volume

Photon Controller includes precreated flavors for persistent disk. The flavors vary by type of storage, such as NFS and VMware vSAN.

photon flavor list
2684...  service-vsan-persistent-disk         persistent-disk  storage.VSAN 1 COUNT
3571...  service-nfs-persistent-disk          persistent-disk  storage.NFS 1 COUNT
395a...  service-shared-vmfs-persistent-disk  persistent-disk  storage.SHARED_VMFS 1 COUNT
812f...  service-local-vmfs-persistent-disk   persistent-disk  storage.LOCAL_VMFS 1 COUNT
67a5...  service-generic-persistent-disk      persistent-disk  persistent-disk 1 COUNT

Creating a Static Persistent Disk for Use in a Pod

You can use a flavor to create a static persistent disk in Photon Controller:

photon disk create --name demo-disk \
                   --flavor service-shared-vmfs-persistent-disk \
                   --capacityGB 2

Check your working by listing the disks:

photon disk list
ID                                    Name         State
e8dbc38c-9374-4099-8a09-fcf4be0a7641  shared-disk  DETACHED

Create a pod in Kubernetes that uses the statically provisioned persistent-disk:

kubectl apply -f nginx-pod-static.yaml

Here is the content of nginx-pod-static.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-demo
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
      hostPort: 80
    volumeMounts:
    - name: photon-storage
      mountPath: /data
  volumes:
  - name: photon-storage
    photonPersistentDisk:
      pdID: "e8dbc38c-9374-4099-8a09-fcf4be0a7641”

Finally, verify that it's attached:

photon disk list
ID                                    Name         State
e8dbc38c-9374-4099-8a09-fcf4be0a7641  shared-disk  ATTACHED