Write a basic Dotmesh Operator that replicates our current DaemonSet setup

[alaric-dotmesh]

This is part of epic #385 .

With #343 done, we can write a simple Dotmesh Operator that runs Dotmesh on every node in the cluster.

This can become the canonical way of running DM in k8s once documented!

[lukemarsden]

Per #343 (comment), we don't have a StatefulSet template per se, but we do have a design for an operator which will set up PVCs, node labels etc appropriately!

[lukemarsden]

As part of this issue, we need a testing strategy.

We are going to develop simple dind-flexvolume and dind-dynamic-provisioner modules which can be used to simulate cloud provider volumes (insofar as they provide writeable filesystems which dotmesh can initialize zfs pools-in-files on) in the dind tests.

Then we can write dind tests for the dotmesh operator creating pods which consuming PVCs from a dind storageclass and providing dotmesh PVs.

Later, we'll be able to simulate killing a dotmesh node and having the PV failover and "reattach" to a different dind.

[lukemarsden]

We now have dind flexvolume and dynamic provisioners, which work according to this test.

Next steps, as I see them:

• write a test that cordoning a node results in the fake "block device" for a pod being reattached by Kubernetes to the pod on the new node

LOCAL

• write a test that providing the dotmesh operator (which is partially implemented here), works just as well the current daemonset. make it pass with the following config by implementing enough of the algorithm.

storageMode: local
localMode:
  poolSizePerNode: 10G
  poolLocation: /var/lib/dotmesh

PV PER NODE

• write a test that providing the dotmesh operator with the following config provisions PVs from the dind provisioner. make it pass by implementing more of the algorithm:

storageMode: pvPerNode
pvPerNodeMode:
  pvSizePerNode: 100G
  storageClass: fast

• write a test which deletes a dind node and demonstrates that the dotmesh operator fails over the PV to another node. make it pass by implementing more of the algorithm. you'll need to support multiple zpools on a single node (by running two dotmesh storage instances on one node).

POOL OF DOTMESHES

• implement NFS support... and then write a test for the following config where storage nodes are only a subset of the nodes in the cluster, demonstrating accessing a dot from a different node to where the storage is hosted:

storageMode: storageNodes
storageNodesMode:
  storageNodes: 3
  sizePerNode: 100G

[lukemarsden]

Note that some of the above plan spans different github issues!

[lukemarsden]

In particular "pool of dotmeshes" depends on #341, #345 and #346 and the unissued final bullet in #100

[lukemarsden]

see https://kubernetes.io/blog/2018/01/introducing-client-go-version-6 Updating dependencies – golang/dep

[alaric-dotmesh]

I'm moving the "extra work" out into a new epic which this is just the first part of. This is now part of epic #385, which is a sub-epic of #100!

[alaric-dotmesh]

This is in production, so I'm calling it done.