Endpoint changes with k8s 1.5

[ravishivt]

In kubernetes 1.5, the "parameters" options in the storage class no longer use the "endpoints" parameter. As per kubernetes/kubernetes#34705:

When the persistent volumes are dynamically provisioned, the Gluster plugin automatically create an endpoint and a headless service in the name `gluster-dynamic-

Leaving it in will generate the below error when trying to create the PVC. This option needs to be removed from the Hello World storage class example. Also, gk-deploy should not even create the "heketi-storage-endpoints" at all since k8s handles the endpoint.

ubuntu@kube1:~$ kubectl describe pvc/gluster1
Name:           gluster1
Namespace:      default
Status:         Pending
Volume:
Labels:         <none>
Capacity:
Access Modes:
Events:
  FirstSeen     LastSeen        Count   From                            SubobjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                            -------------   --------        ------                  -------
  10m           14s             41      {persistentvolume-controller }                  Warning         ProvisioningFailed      Failed
to provision volume with StorageClass "gluster-heketi": glusterfs: invalid option "endpoint" for volume plugin kubernetes.io/glusterfs

[screeley44]

It will be removed with when OCP 3.5 is released or we could even have version specific defs

[ravishivt]

Here's an example of k8s automatically creating the endpoint:

ubuntu@kube1:~$ sudo kubectl create -f docs/examples/hello_world/gluster-pvc.yaml
persistentvolumeclaim "gluster1" created
ubuntu@kube1:~$ sudo kubectl get endpoints
NAME                         ENDPOINTS                                      AGE
glusterfs-dynamic-gluster1   172.16.35.11:1,172.16.35.12:1,172.16.35.13:1   10s
heketi                       192.168.149.139:8080                           23h
heketi-storage-endpoints     172.16.35.11:1,172.16.35.12:1,172.16.35.13:1   23h
kubernetes                   172.16.35.11:6443                              2d

[ravishivt]

@screeley44 OCP? Sorry, not familiar with that.

[screeley44]

OpenShift Container Platform - I don't think Kubernetes 1.5 is officially released yet is it? OCP is always behind Kube as that is downstream and Kube is upstream, so we might have to manage those differences in some way, I guess we also need an OCP version of HelloWorld doc as well

[ravishivt]

OK, thanks. Familiar with OpenShift, just not that acronym. Kubernetes v1.5 was just released a couple days ago.

[jarrpa]

We do need to maintain compatibility with both Kubernetes and OpenShift. This will require a few changes in various places, starting with https://github.com/heketi/heketi. The setup-openshift-heketi-storage command of heketi-cli will need to be trimmed down so it only creates the DB secret and the copy job. Then we'll still need to create the endpoints manually for OpenShift and also Kube < 1.5 (which will require a version check). Once these things are done we should be able to get away with not creating them for Kube 1.5.

@ravishivt Would you be willing to open the issue on the heketi project, as a compatibility request for Kubernetes 1.5?

[ravishivt]

Sure, heketi/heketi#628

[jarrpa]

Since this PR was last updated, it has been determined that the heketi-storage-endpoints are still required since the heketidbstorage volume is statically provisioned. Further, the Hello World example has been updated to indicate that endpoints: should be omitted from Kube >= 1.6.