Raft cluster on K8s/Openshift and multiple K8s cluster

[askids]

hi,

I had a couple of questions on the usage, when hosting on K8s./Openshift. Lets say my Asp.Net Core API is hosted on Openshift, So when trying to join a pod instance to the RAFT cluster, do we need to still join the cluster using IP and port only or is there alternative option? Also since IPs will be random assigned, whenever the pod comes up, how will a specific instance know which cluster join? It would be helpful, if we have specific examples around usage on K8s.

Also, if API is hosted on 2 different K8s cluster, is it possible to join a single Raft cluster as the IP of the pods will be internal to that K8s/Openshift cluster. Typically for outside Openshift cluster access, we use routes which gives a load balanced URL. So behind the route, based on my deployment configuration, I could be running 1 pod, 2 pod or 10 pods. So in such cases, how do I ensure that every pod instance joins the Raft cluster?

Thanks!

[sakno]

There is no one true way to use .NEXT Raft in K8s environment. Instead, there are many ways much or less convenient in your circumstances. I can provide a quick overview of how it was applied in one of the companies that uses the library.

1. First of all, you need to understand that a new node requires bootstrapping procedure. In other words, it should be warmed up before accepting external traffic. It means that the node must be in sync with the rest of cluster. Otherwise, the client request will wait for a long time for sync because of read barrier (to provide linearizable read). Therefore, Ingress controller should not distributed the traffic to the node while it is warming up.
2. To achieve that, you can use K8s startup probe. It is built-in mechanism of microservices with (relatively) long initialization time. As an option, you can use .NEXT AMI to implement various probes.
3. The node must start with coldStart=false configuration property according to the documentation. Then, you can use IRaftCluster.Readiness property to implement the probe (and wait for full replication).
4. When a new node is started, you need to inform the leader to modify cluster configuration and apply changes to the membership list. To do so, .NEXT Raft has announcer as extensibility point. You need to register ClusterMemberAnnouncer<UriEndPoint> delegate instance in DI container.
5. Announcer can be implemented using internal HTTP endpoint which is available within the cluster only (but not externally). Configuration changes are allowed on leader node only. You can organize your nodes in the cluster using Service abstraction. Then, your announcer can enumerate all nodes in the cluster using this abstraction and find the leader (or use automatic message routing provided by the library) to announce a new node.
6. You can use K8s client for .NET to communicate with K8s control plane

Or you can implement background service as a part of your node and poll a list of nodes through Service abstraction and compare the list with the actual list of cluster members. If there are changes, just apply them to the configuration if the current node is a leader node.

Another way is to implement K8s Operator.

[sakno]

One more thing - you can host Raft-related stuff in WebHost separately from the main application's WebHost and use different port.

[wux5]

In the worst scenario, it can switch Leader node many times when deploying new build to k8s. Let's say we have 5 nodes in the cluster, k8s can deploy to the leader node first which will make the cluster select a new Leader, k8s finish deploying first node and now move on to next which happen to be the new Leader, and keep being unlucky in this way and switch Leader node 5 times for the whole process. In our case switching Leader node will have some negative impact due to the replication process, is there a more graceful way for the deployment so that it need to change Leader only once?

Is there a way we can somehow influence the leader selection algorithm to choose a node that has been just upgraded? thanks @sakno

[sakno]

Adding new node doesn't force re-election. coldStart=true is only needed when starting the first (and single) node in the cluster. While cluster has at least one node, you don't need to start anything with that parameter. I meant that any manipulations with membership can be done on the leader node only.

Also, you could skip coldStart at all and bootstrap the cluster with pre-populated configuration consisting of 3 nodes.

[wux5]

Thanks. I guess my problem is not about starting new nodes, but after all nodes are started and now we need to re-deploy, which will gradually replace the nodes one by one. Yes it can always re-elect a new leader, my problem is, I want to avoid re-elect new leader multiple times during the deployment process.

[sakno]

If re-deploy implies node restart then yes, re-election is unavoidable.