etcdserver: add ability to auto-promote learners to voters

[maxenglander]

Auto-promote learners

See #10537 for original mention of auto promoting learners.
See etcd-io/website#107 for changes to learner design doc.

Design is motivated by comments in original design PR:

There are three cases for auto-promote:

• promote a learner to voter regardless of the cluster size.
• promote a learner to voter if the cluster size is smaller than the expected size (for example, a voter is removed from the cluster)
• promote a learner to voter if some other voter is unhealthy for some user defined duration (remove the unhealthy node and replace it with a learner)

...and also motivated by other planned 3.5 learner features:

A learner can serve as a read-only node that never gets promoted. In a weak consistency mode, learner only receives data from leader and never process writes. Serving reads locally without consensus overhead would greatly decrease the workloads to leader but may serve stale data. In a strong consistency mode, learner requests read index from leader to serve latest data, but still rejects writes.

Enables newly added learners to be automatically promoted to voting members.

With this PR, operators can supply "auto" and "promote rules" to member add API. When --learner and --auto are supplied to member add, the learner is automatically promoted to a voting member when any of its promotion rules are satisfied. If no promotion rules are supplied, the "default" promotion rule is used (90% caught up to leader progress).

If --learner is supplied but not --auto, the learner is not automatically promoted. Instead, the promotion rules determine whether an operator request to promote the learner is accepted. This implementation supplants and maintains compatibility with the current --learner behavior.

Operators can also supply --delay, which determines how long a promotion rule must be satisfied before voter is promoted. The default value is 0, meaning learners may be promoted immediately when a promotion rule is satisfied.

With this PR, learners may only be promoted to voters. However, the API changes are flexible enough to accommodate other promotion targets (e.g. "strong reader" or "weak reader"). Likewise, with this PR the only promotion criteria is "--min-progress". The API can easily accommodate additional promotion criteria such as --min-healthy-voters (or --min-cluster-size), which would promote member as soon as the number of healthy voters (or number of members) in cluster drops below a threshold.

[jingyih]

Thanks for implementing this feature. The original plan was to not include this in 3.4 release (will likely be in 3.5), but I am open to discussion on this. FYI, the feature cut date for 3.4 release is July 31, accordingly to the latest community meeting.

cc @xiang90 @gyuho @jpbetz @WIZARD-CXY

[jingyih]

Is this still WIP? Is it ready to be reviewed?

[maxenglander]

Hey @jingyih it's WIP. I still need to write tests, update documentation and some APIs (e.g. member list to include info on auto-promotion). I was hoping to get some initial feedback on the approach while I'm working on those things.

[WIZARD-CXY]

Thanks @maxenglander. I will take a closer look asap.

[WIZARD-CXY]

@maxenglander I went throught the code change, the overall looks good. Can we keep the autoPromote related code in the etcdserver?

[maxenglander]

@WIZARD-CXY are you asking if I can undo changes to all packages except etcdserver? If I do that, it won't be possible to provide users with an --auto-promote flag.

[jingyih]

We need to decide whether autoPromote is an etcd feature or Raft feature. I don't think changes to Raft package is necessary for implementing this feature. I prefer to keep Raft package having core functionalities only - implement autoPromote only in etcd server.

[jingyih]

I think we should keep MemberAdd() unchanged for better backward compatibility.

[maxenglander]

Good point, done in 29fe958

[jingyih]

For better readability, maybe use a flag to record if this member is promoted, so that you can just do 'break' after this for loop.

[maxenglander]

Good idea, done in 29fe958

[maxenglander]

Hey @jingyih I can make that change, but could benefit from some input on how. I can see two ways of implementing this without changes to Raft package.

1. Change the behavior of member add --learner so that all learners are auto-promoted
2. Keep track of whether a learner is designated for auto-promotion only on the leader (in memory and/or on disk)

The drawback of 1. is that it is a breaking API change. Users might expect learners to remain in that state until manually promoted.

The drawback of 2. is that if the leader dies and leadership transfers to another member, a learner that is marked for auto-promotion will lose its auto-promotion designation, and thus be "demoted" to a regular learner, unless there is some (non-Raft) mechanism for having leaders share auto-promote designations with peers.

Can you provide input on whether 1. or 2. is preferable? Or perhaps some 3rd option I'm not thinking of?

I'd also be interested in understanding what criteria you use to decide that the ability to have add a learner qualifies as a Raft feature, whereas the ability to designate a learner for auto-promotion does not.

[jingyih]

The first step we should do is probably decide the behavior of member add API in v3.5. My own opinion is, in the very long run, 'member add' by default may simply add any new member as learner, which will be auto promoted once caught up. If user wants to add a learner member that will not be auto promoted, they need to specially set a flag. But in the near future (v3.5), the API is not clearly defined yet. (We might also need to implement a proper feature gate and put auto promote behind it) I am happy to start a draft on this so we could start the discussion.

I can make that change, but could benefit from some input on how. I can see two ways of implementing this without changes to Raft package.

I'd also be interested in understanding what criteria you use to decide that the ability to have add a learner qualifies as a Raft feature

Sorry my previous comment is inaccurate. What I meant was, auto promote's implementation might not need changes to Raft package and current API. etcdserver has cluster member information in s.cluster. So one possible option is to add autoPromote field only in etcdserver. From Raft's perspective, it may not need to distinguish between the learners. - I am happy to provide more details and suggestion. But since I feel the actual implementation is going to be depend on member add API in v3.5, we should agree on the API first.

[maxenglander]

@jingyih thanks for the comments. I'm happy to wait for you and other maintainers to firm up API plans after which I'm happy to make adjustments to code based on that. FWIW, here was what I was envisioning for API changes from v3.4 and onwards.

My own opinion is, in the very long run, 'member add' by default may simply add any new member as learner, which will be auto promoted once caught up.

I agree, I think this makes a lot of sense. My thinking was that, in v3.4, users would have the option of specifying member add --learner --auto-promote as a way of previewing that "long run" behavior.

In >= v3.5, I was thinking member add would end up behaving the same way as member add --learner --auto-promote, which would mostly be a matter of changing member add to trigger a pb.ConfChangeAddAutoPromotingNode. At that time, pb.ConfChangeAddNode would become an internal-only configuration change that can not be triggered via user API call, and is meant to be triggered by etcdserver once an auto-promoting node has caught up to leader.

Eventually (e.g. in v3.6 or later), member add --learner --auto-promote could optionally be deprecated since it would be redundant to member add.

If user wants to add a learner member that will not be auto promoted, they need to specially set a flag.

I was thinking that --learner would be that special flag. In other words, the behavior of member add would change starting at v3.5, but member add --learner would remain the same.

So one possible option is to add autoPromote field only in etcdserver. From Raft's perspective, it may not need to distinguish between the learners.

Let me know if I'm understanding you correctly. I think you're saying that autoPromote could be a global value set by a feature-gate, rather than (as in current PR) something that is specifiable per-learner. In other words, a given etcd cluster could support either auto-promoting learners or non-auto-promoting learners, but not both. If it were implemented that way, then pb.ConfChangeAddLearner would create an auto-promoting learner or a non-auto-promoting learner, depending on the value of the feature gate known in etcdserver, and therefore there would be no need for pb.ConfChangeAddAutoPromotingNode.

Thanks for all the discussion so far. As I said above, will wait to see what you and other members decide on in terms of v3.5 API, and make code adjustments accordingly.

[jingyih]

@maxenglander I agree with most part of your envisioning. Let's first finalize the API. Maybe just open a PR against the design doc [1] to add a new section 'Learner Implementation in v3.5'?

At the same time, I would like to explore the opportunity of possibly adding proper feature gate in etcd. AFAIK, currently in etcd the experimental features are enabled by special server side flags [2]. If not being careful, servers in the same cluster could end up having different settings on these. If we can improve this situation, we can choose to (or not to) put auto promote behind a feature gate, in additional to the '--auto-promote' flag in member add.

Another nice thing about having feature gate, is that we can always disable a new feature when it is added, and enable it in the next minor release. This is one step further towards officially support 1 minor version downgrade / rollback.

[1] https://github.com/etcd-io/etcd/blob/master/docs/server-learner.rst

[2] https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/configuration.md#experimental-flags

[maxenglander]

That sounds good @jingyih I will do that. Thank you for the guidance.

[codecov-io]

Codecov Report

Merging #10887 (fdb1712) into master (b67ed4e) will decrease coverage by 22.61%.
The diff coverage is 31.57%.

@@             Coverage Diff             @@
##           master   #10887       +/-   ##
===========================================
- Coverage   69.74%   47.12%   -22.62%     
===========================================
  Files         407      398        -9     
  Lines       32585    32614       +29     
===========================================
- Hits        22726    15370     -7356     
- Misses       7895    15220     +7325     
- Partials     1964     2024       +60     

Flag	Coverage Δ
all	47.12% <31.57%> (-22.62%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
client/v3/cluster.go	18.18% <0.00%> (-74.32%)	⬇️
server/etcdserver/api/etcdhttp/peer.go	40.27% <0.00%> (-46.87%)	⬇️
server/etcdserver/api/membership/cluster.go	56.92% <0.00%> (-23.25%)	⬇️
server/etcdserver/api/membership/errors.go	100.00% <ø> (ø)
server/etcdserver/api/v2v3/server.go	0.00% <0.00%> (ø)
server/etcdserver/cluster_util.go	40.14% <0.00%> (-28.24%)	⬇️
server/etcdserver/errors.go	0.00% <ø> (ø)
server/etcdserver/util.go	65.42% <0.00%> (-33.21%)	⬇️
server/proxy/grpcproxy/cluster.go	30.92% <10.00%> (-39.53%)	⬇️
etcdctl/ctlv3/command/member_command.go	61.83% <25.00%> (-5.41%)	⬇️
... and 253 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b67ed4e...fdb1712. Read the comment docs.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[maxenglander]

/remove stale

[maxenglander]

/remove-stale

[maxenglander]

Hi @ptabor thanks for reviewing etcd-io/website#107. I thought since you reviewed that I would ping you here to see if you can help move this PR forward. There are probably some additional changes I need to make in this PR, but I think it's a point where it can be reviewed.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[maxenglander]

Hi @ptabor @jingyih can we re-open this? I didn't mean to let it close.

[maxenglander]

/remove-stale

[lilic]

@maxenglander I reopened, do you mind rebasing with the main branch, thanks!

[maxenglander]

Thank you for re-opening @lilic! I've rebased.

Ping @jingyih @ptabor let me know if there's a good way to move this PR forward. I'm still interested in getting this merged.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.