You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/server-learner.rst
+30-30Lines changed: 30 additions & 30 deletions
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,8 @@
1
-
.. _server-non-voting-member:
1
+
.. _server-learner:
2
2
3
3
4
-
Non-voting Member
5
-
#################
4
+
Learner
5
+
#######
6
6
7
7
8
8
:Authors:
@@ -19,93 +19,93 @@ Membership reconfiguration has been one of the biggest operational challenges. L
19
19
20
20
A newly joined etcd member starts with no data, thus demanding more updates from leader until it catches up with leader’s logs. Then leader’s network is more likely to be overloaded, blocking or dropping leader heartbeats to followers. In such case, a follower may election-timeout to start a new leader election. That is, a cluster with a new member is more vulnerable to leader election. Both leader election and the subsequent update propagation to the new member are prone to causing periods of cluster unavailability (see *Figure 1*).
What if network partition happens? It depends on leader partition. If the leader still maintains the active quorum, the cluster would continue to operate (see *Figure 2*).
What if the leader becomes isolated from the rest of the cluster? Leader monitors progress of each follower. When leader loses connectivity from the quorum, it reverts back to follower which will affect the cluster availability (see *Figure 3*).
When a new node is added to 3 node cluster, the cluster size becomes 4 and the quorum size becomes 3. What if a new node had joined the cluster, and then network partition happens? It depends on which partition the new member gets located after partition. If the new node happens to be located in the same partition as leader’s, the leader still maintains the active quorum of 3. No leadership election happens, and no cluster availability gets affected (see *Figure 4*).
If the cluster is 2-and-2 partitioned, then neither of partition maintains the quorum of 3. In this case, leadership election happens (see *Figure 5*).
What if network partition happens first, and then a new member gets added? A partitioned 3-node cluster already has one disconnected follower. When a new member is added, the quorum changes from 2 to 3. Now, this cluster has only 2 active nodes out 4, thus losing quorum and starting a new leadership election (see *Figure 6*).
Since member add operation can change the size of quorum, it is always recommended to “member remove” first to replace an unhealthy node.
57
57
58
58
Adding a new member to a 1-node cluster changes the quorum size to 2, immediately causing a leader election when the previous leader finds out quorum is not active. This is because “member add” operation is a 2-step process where user needs to apply “member add” command first, and then starts the new node process (see *Figure 7*).
An even worse case is when an added member is misconfigured. Membership reconfiguration is a two-step process: “etcdctl member add” and starting an etcd server process with the given peer URL. That is, “member add” command is applied regardless of URL, even when the URL value is invalid. If the first step is applied with invalid URLs, the second step cannot even start the new etcd. Once the cluster loses quorum, there is no way to revert the membership change (see *Figure 8*).
Same applies to a multi-node cluster. For example, the cluster has two members down (one is failed, the other is misconfigured) and two members up, but now it requires at least 3 votes to change the cluster membership (see *Figure 9*).
As seen above, a simple misconfiguration can fail the whole cluster into an inoperative state. In such case, an operator need manually recreate the cluster with ``etcd --force-new-cluster`` flag. As etcd has become a mission-critical service for Kubernetes, even the slightest outage may have significant impact on users. What can we better to make etcd such operations easier? Among other things, leader election is most critical to cluster availability: Can we make membership reconfiguration less disruptive by not changing the size of quorum? Can a new node be idle, only requesting the minimum updates from leader, until it catches up? Can membership misconfiguration be always reversible and handled in a more secure way (wrong member add command run should never fail the cluster)? Should an user worry about network topology when adding a new member? Can member add API work regardless of the location of nodes and ongoing network partitions?
77
77
78
78
Raft Learner
79
79
============
80
80
81
-
In order to mitigate such availability gaps in the previous section, `Raft §4.2.1 <https://ramcloud.stanford.edu/~ongaro/thesis.pdf>`_ introduces a new node state “Learner”, which joins the cluster as a non-voting member until it catches up to leader’s logs.
81
+
In order to mitigate such availability gaps in the previous section, `Raft §4.2.1 <https://ramcloud.stanford.edu/~ongaro/thesis.pdf>`_ introduces a new node state “Learner”, which joins the cluster as a **non-voting member** until it catches up to leader’s logs.
82
82
83
83
Features in v3.4
84
84
----------------
85
85
86
86
An operator should do the minimum amount of work possible to add a new learner node. ``member add --learner`` command to add a new learner, which joins cluster as a non-voting member but still receives all data from leader (see *Figure 10*).
When a learner has caught up with leader’s progress, the learner can be promoted to a voting member using ``member promote`` API, which then counts towards the quorum (see *Figure 11*).
etcd server validates promote request to ensure its operational safety. Only after its log has caught up to leader’s can learner be promoted to a voting member (see *Figure 12*).
Learner only serves as a standby node until promoted: Leadership cannot be transferred to learner. Learner rejects client reads and writes (client balancer should not route requests to learner). Which means learner does not need issue Read Index requests to leader. Such limitation simplifies the initial learner implementation in v3.4 release (see *Figure 13*).
In addition, etcd limits the total number of learners that a cluster can have, and avoids overloading the leader with log replication. Learner never promotes itself. While etcd provides learner status information and safety checks, cluster operator must make the final decision whether to promote learner or not.
0 commit comments