Added heketi-turnkey deployment

[lenartj]

This is a proof of concept deployment using only kubernetes objects (and a docker image, see below).
https://asciinema.org/a/103212

While it is useful for me I am not sure how useful it is in general :-)
It solves issue #161

---

This change is 

[jarrpa]

I'm still relatively new to containers, so please help me understand how this works. You would use the included Dockerfile to build a container, and then deploy the ConfigMap and the Pod (which uses the container you just built) which executes /opt/gk-deploy from inside the container. How does gk-deploy know what Kubernetes cluster it's working against, and how does it have administrative access to the cluster?

[lenartj]

Hi jarrpa,

The Docker image is an automatic build available on docker hub as lenart/heketi-turnkey. (If you want you can build your own image using that Dockerfile. In that case you will need to download the heketi-turnkey.yaml file and modify it to point to your image.)

It works on the kubernetes cluster you create the pod on. It uses https://kubernetes as the api server url. (A service named 'kubernetes' already exists in all kubernetes clusters pointing to the api server and kube-dns makes it possible for pods to resolve that name to the correct IP address of the master/api server pod.)

The pod is configured to use the default service account in the default namespace. The level of access that service account provides depends on your cluster deployment. If that does not have the needed role and you are not happy to assign it then you may create a new service account, assign the needed access to that and then modify the heketi-turnkey.yaml file to use that service account (the spec.serviceAccount field).

(Apologies for not providing too much documentation yet ... I wanted to get it out quick)

[harshal-shah]

Hi @lenartj ,
I'm now able to setup gluster on my k8s cluster, but I cant create any persistent volume claims because when I provide the resturl of heketi for storage class, it is not reachable.
I think this could be because kubernetes is trying to reach rest-url of heketi service externally which cannot work.

• heketi service

NAME                       CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
heketi                     10.39.242.44   <none>        8080/TCP   50m
heketi-storage-endpoints   10.39.245.92   <none>        1/TCP      50m
kubernetes                 10.39.240.1    <none>        443/TCP    1h
postgres-svc               None           <none>        5432/TCP   14m

• storage class definition

kind: StorageClass
metadata: 
  name: gluster-heketi
provisioner: kubernetes.io/glusterfs
parameters:
    resturl: "http://10.39.242.44:8080"
    restuser: "joe"
    restuserkey: "My Secret Life"

• pvc definition

kind: PersistentVolumeClaim
metadata:
  name: gluster1
  annotations:
    volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

• Error in pvc

kubectl describe pvc gluster1
Name:           gluster1
Namespace:      default
StorageClass:   gluster-heketi
Status:         Pending
Volume:
Labels:         <none>
Capacity:
Access Modes:
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                            -------------   --------        ------                  -------
  1h            26s             92      {persistentvolume-controller }                  Warning         ProvisioningFailed      Failed to provision volume with StorageClass "gluster-heketi": glusterfs: create volume err: error creating volume Post http://10.39.242.44:8080/volumes: dial tcp 10.39.242.44:8080: i/o timeout.

[lenartj]

Hi @harshal-shah,

Could you post the logs of the heketi pod?
kubectl logs $(kubectl get pod -l glusterfs=heketi-pod -o name | cut -d / -f2)

Could you try removing the restuser and restuserkey parameters from your storageclass?

Have you got the dm_thin_pool kernel module loaded and the glusterfs-client package installed on your nodes?

[harshal-shah]

Hi,

• Here is the output of the pod:

$ kubectl logs $(kubectl get pod -l glusterfs=heketi-pod -o name | cut -d / -f2)
Heketi v4.0.0-4-g5d702fe
[heketi] INFO 2017/02/20 07:28:32 Loaded kubernetes executor
[heketi] INFO 2017/02/20 07:28:32 Loaded simple allocator
[heketi] INFO 2017/02/20 07:28:32 GlusterFS Application Loaded
Listening on port 8080

• This output is generated by removing the restuser and restuserkey parameters from storage class

• Yes, I have loaded the dm_thin_pool kernel parameter and have ensured glusterfs-client v3.6.9 is present on all my nodes

[lenartj]

Could there be a problem with your cluster network?
Could you try executing curl http://10.39.242.44:8080/hello from each of your nodes?

Also, can you try kubectl get endpoint heketi to see if the heketi pod is picked up the the heketi service?

[harshal-shah]

The service is accessible from nodes :

$ wget -qO- http://10.39.242.170:8080/hello
Hello from Heketi

Also the pod is picked up by the service,

$ kubectl get endpoints heketi
NAME      ENDPOINTS        AGE
heketi    10.36.0.3:8080   4m

Above results are from a new cluster which I have spun up. I had deleted old cluster yesterday night.

[harshal-shah]

For now, I managed to proceed ahead by making heketi service of type LoadBalancer. Thanks for your help

[lenartj]

@harshal-shah, are you running on GKE or AWS or ...? I am happy that it works :-) There should be no need for a LoadBalancer in front of this service though so I suggest you keep investigating what's odd. At this point I'm almost certain it's a network setup issue.

[harshal-shah]

@lenartj yes, I'm using GKE. As we already found that the service URL works from nodes, I will try to open firewall to all IP's to check if that is the issue. Will keep you updated.

[lenartj]

Oh, I understand, so you need to be able to provision volumes from outside the cluster. Keep in mind that LoadBalancer is not free on GKE, comes out to be min. ~$20/month/lb. You could make the Service type NodePort for example.

[harshal-shah]

Yes, I was thinking of that as well. Making the service as NodePort seems like the best option.

Volume provisioning within the cluster didn't work for me as well.

[kwtalley]

I've actually run into the same problem as @harshal-shah :
glusterfs: create volume err: error creating volume Post http://<heketi_clusterip>:8080/volumes: dial tcp <heketi_clusterip>:8080: i/o timeout.

even worse though, i cannot curl clusterip:8080/hello, it times out as well. i can curl /hello on the service endpoint though from 'kubectl get endpoints heketi'

i'm guessing also that i have some network issues?

[harshal-shah]

Another successful way that worked for me was exposing the heketi service as a NodePort (thanks @lenartj ) and mention public IP of any host + Node Port in storage class definition.
I fail to understand why kubernetes fails to create the volumes using the heketi service cluster IP?
If I spin up a busybox pod, then the heketi service cluster IP is accessible, IMO kubernetes should be able to create cluster using service IP without need for exposing NodePort.

Is it possible that the glusterfs provisioner is calling heketi service from outside of kubernetes network rather than from within ?

[lenartj]

@harshal-shah, @kwtalley, thanks a lot for testing.. I can see the problem now. You are both running on GKE, and on GKE the master is outside of the cluster. The provisioning requests are originated from the apiserver, which is hosted by Google for you.

1. I've created a dummy storageclass on a test cluster (1.2.3.4 is the public ip of random server of mine on the internet):

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: heketi
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://1.2.3.4:2020/"

2. On 1.2.3.4 I did: nc -vvvv -l -p 2020

3. I've created a pvc on the cluster for a 4GB volume

4. Voila

$ nc -vvvv -l -p 2020
listening on [any] 2020 ...
connect to [1.2.3.4] from 226.11.187.35.bc.googleusercontent.com [35.187.11.226] 33368
POST //volumes HTTP/1.1
Host: 1.2.3.4:2020
User-Agent: Go-http-client/1.1
Content-Length: 133
Authorization: bearer REDACTED
Content-Type: application/json
Accept-Encoding: gzip

{"size":4,"name":"","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"snapshot":{"enable":false,"factor":0}}

35.187.11.226 was the IP of the Kubernetes apiserver hosted by Google.

So the cluster IPs are not reachable from that apiserver that's why it's worked for you when you created a LoadBalancer or a NodePort and referred to the public ip of a node.

[harshal-shah]

@lenartj Can you host one more image version of image lenart/heketi-turnkey with tag NodePort which exposes the heketi service as NodePort ? If not, I will host it under my docker hub user.

[kwtalley]

well i've been digging into this more and haven't made much progress. my setup is actually a bare-metal cluster in my own lab. for some reason i can only reach a service ip from the node hosting the service pod. the other nodes in the cluster cannot reach it. been searching in iptables rules and getting some tcp dumps and seeing the following:
node A: curl serviceIP:8080/hello
node B (hosting service pod): request received on flannel interface from node A's host interface, instead of node A's flannel interface.
packet dropped.
all the rules in iptables on both nodes look correct with all SNAT and DNAT entries...still lost on this

[kwtalley]

turns out kube-proxy did not have the --cluster-cidr=flannel-network/mask entry on the worker nodes. adding that fixed the services! @harshal-shah check if this is set on your workers kube-proxy config

[jarrpa]

Looking over this again, I'm not sure that this is a better solution than #155, which seems more robust, less fiddly, and better able to handle errors. The only downside I see to the other solution is requiring ansible as a dependency. @lenartj can you make an argument as to the pros of this solution over the other one?

[jarrpa]

Closing this PR. It can be reopened if requested.

[blublinsky]

This is extremely useful.
Is there a plan to make it officially supportable part of the project?

[jarrpa]

Nope, still isn't.