Use etcd3 instead of GlusterFS volume for heketi DB

[jarrpa]

When heketi/heketi#599 resolves, use an etcd3 cluster (running in kube) to store the heketi DB instead of a direct BoltDB in a GlusterFS volume. This should remove the need for an initial deploy-heketi pod. We could use this as reference.

[ant31]

@jarrpa what will be the volume for the etcd3 ? hostPath? emptyDir?

[jarrpa]

It will be hostPath, I believe, and the etcd3 service takes care of the synchronization across the nodes. Likely somewhere in /var/lib/heketi.

[joejulian]

As also mentioned in the associated heketi issue, as of kubernetes 1.6, the suggested store is configmap instead of going directly to etcd3.

[jarrpa]

This is mostly obsolete since heketi implemented the ability to backup to a Secret. Closing.