add support in haketi to allow nodes to be in two different cns clusters

[ekuric]

Currently , heketi supports nodes to be only in one CNS cluster, that means it is not possible to build structure like

• cns cluster 1 : nodes: node1, node2, node3, and devices /dev/sdb, /dev/sdc, /dev/sdd on these nodes
• cns cluster 2 : nodes: node1, node2, node3, and devices /dev/sdf, /dev/sdg, /dev/sdp on these nodes

idea is for nodes to host two different ( or more ) clusters where clusters are using different block devices on nodes

[MohamedAshiqrh]

@ekuric Yeah, this will be useful if they have both ssd and hdd in same node. @lpabon @obnoxxx Along with the --cluster option we can have a label --device-set "some label" which can be used to choose faster/notsofast devices. What do you guys think?

[obnoxxx]

@MohamedAshiqrh I think the label request is actually an independent thing (which I want to detail in a separate issue).

@ekuric Maybe I have not fully understood your suggestion. But there is no such thing as a CNS cluster. There are gluster clusters, and gluster can afaik only have a given node participate in one single cluster. But referring to CNS: CNS is the solution that lets heketi and gluster run as pods (containers) in kubernetes/openshift. There you might want to add more gluster clusters to the same physical hosts for scaling purposes: There is currently a hard limit at around 100 volumes per cluster (due to glusterd). Hence in order to support more than this number of volumes with some beefy hardware, you would have to deploy multiple gluster clusters on the same machines. But this is not a problem, since in kubernetes, gluster nodes are running as containers, and afaict nothing prevents you from putting multiple gluster clusters on the same kubernetes nodes.

[ekuric]

@obnoxxx yes,you are right, please forget cns as it is "downstream", by mistake I used that term.

We talk here about heketi cluster. I do not think we can run multiple gluster clusters on same hardware / nodes ( at current status and when running in kubernetes/openshift ) it will fail to start it due to [1]

[1]

#  heketi-cli node list

Id:6ff945f8072e961479e293dee6e0f44b
Cluster:5204854165a52a59e7cf062d40df577d

Id:ca72cadab072a51cca9e90f6ffe81332
Cluster:5204854165a52a59e7cf062d40df577d

Id:f66271ab915b1c3fd7fc5fdd722bcd64
Cluster:5204854165a52a59e7cf062d40df577d


# heketi-cli cluster create
Cluster id: 95cfe03d335f160619d3f3eaaf350c24

# heketi-cli cluster list

Clusters:
5204854165a52a59e7cf062d40df577d
95cfe03d335f160619d3f3eaaf350c24

first cns cluster :

root@ip-172-31-29-254: ~ # heketi-cli cluster info
5204854165a52a59e7cf062d40df577d
Cluster id: 5204854165a52a59e7cf062d40df577d
Nodes:
6ff945f8072e961479e293dee6e0f44b
ca72cadab072a51cca9e90f6ffe81332
f66271ab915b1c3fd7fc5fdd722bcd64
Volumes:
c657637193d9ae49d58c5dd3615028ce

second cns cluster

root@ip-172-31-29-254: ~ # heketi-cli cluster info
95cfe03d335f160619d3f3eaaf350c24
Cluster id: 95cfe03d335f160619d3f3eaaf350c24
Nodes:

Volumes:

trying to add node to new cluster

root@ip-172-31-29-254: ~ # heketi-cli node add --zone=1
--cluster=95cfe03d335f160619d3f3eaaf350c24
--management-host-name=ip-172-31-19-29.us-west-2.compute.internal
--storage-host-name=172.31.19.29

Error: Hostname ip-172-31-19-29.us-west-2.compute.internal already used by
node with id f66271ab915b1c3fd7fc5fdd722bcd64

it does not allow kubernetes node to be part of two gluster clusters, what prevents to create new gluster cluster running on same hardware/nodes but using different block devices.

[MohamedAshiqrh]

@ekuric @obnoxxx There are (solvable)problems based on hostname/IP to run multiple gluster instances in the same node.
Refer:
http://lists.gluster.org/pipermail/heketi-devel/2017-March/000080.html

[jamie-liu]

I am also facing the same issue as the nodes have both ssd and sata drives. So I want to provide different type of volume based on different hardware.
@ekuric @MohamedAshiqrh I think the issue can be solved by setup two heketi instances which run as container.
And in the storageclass yaml, we can specify different heketi resturl when creating different type of volume.

[phlogistonjohn]

The last activity on this issue is almost a year old. In addition, this seems somewhat outside the scope of heketi and is more an issue with how gluster containers are effectively one-per-node. This seems like a problem better solved at the gluster/GD2 level. Feel free to reopen if more discussion is needed.