Multiple disk with capacity based scheduling

[yasker]

Current Longhorn replicas will use the directory at /var/lib/rancher/longhorn to store the data. We should support multiple user-specified locations on the node to use Longhorn.

We can support the same locations (e.g. in addition to the default location, user can add /opt/storage as the second location, and make sure the directory is available to use on all the nodes) for all the nodes as the first step if we don't need to store node-specific information (hopefully).

In the end, the user can update the node config and set desired disk (mounted to a node directory) to store Longhorn volume data. Longhorn can choose where to place the volume data depends on how much space the disk had left.

[pulberg]

Would be great to use the location where Docker is configured to store images/containers, it is a much larger volume and can be expanded as needed, we currently modify Docker startup with daemon.json -

{
    "graph": "/mnt/dockerdata",
    "insecure-registries": ["blah2.com:7720","blah.com:7721"]
}

[yasker]

Yeah, after we implemented this feature, any arbitrary location on the host would be supported for storing data.

[yasker]

Phase 1, longhorn/longhorn-manager#85 merged.

[yasker]

Done.