etcdserver: add learner metrics

[WIZARD-CXY]

@jingyih @xiang90 Add learner metrics.
I suppose we will get this etcd_debugging_learner_promote_failures and etcd_debugging_learner_promote_success from leader.

[jingyih]

So this counter will only get updated if the local member is leader, right? Consider a user sending promote requests to a server which is not the leader at the moment. Then the user checks metrics, and sees 0 promote failures and 0 promote success. Feels like the description needs to be clearer to avoid user confusion. Maybe 'The total number of learner promotion failures while the server is leader'?

Also there are 3 possible reasons that a promote could fail, maybe we should not suggest it is likely to be one of them?

[WIZARD-CXY]

I get your point. user can use prometheus sum query to get the total number of the failures. As a debugging metrics I think it is fine add failure reasons as the metrics label. @jingyih

[jingyih]

OK.

[WIZARD-CXY]

@jingyih seems I need part3 to be added #10730 in the upstream.Now the relating code is not in the master yet.

[jingyih]

@jingyih seems I need part3 to be added #10730 in the upstream.Now the relating code is not in the master yet.

Maybe just wait for #10730? Hopefully it will get merged soon. BTW KubenCon is starting soon, it might cause some delay in reviews.

[WIZARD-CXY]

@jingyih @xiang90 kindly ping

[gyuho]

Is this still WIP? This PR only contains doc change.

add failure reasons as the metrics label

This will be useful. Can we update this PR with the actual code change?

[gyuho]

We also need changelog (maybe @jingyih can collect all the changes at once in one PR).

[jingyih]

Is this still WIP? This PR only contains doc change.

add failure reasons as the metrics label

This will be useful. Can we update this PR with the actual code change?

This is blocked by #10730.

[jingyih]

We also need changelog (maybe @jingyih can collect all the changes at once in one PR).

I will update the changelog once #10730 is finalized and merged.

[jingyih]

#10730 merged. Please update PR. @WIZARD-CXY

[WIZARD-CXY]

@jingyih will do

[WIZARD-CXY]

@jingyih I check the code and find that it is hard to monitor whether the member is a learner or not real time. The code now is not ready for this yet. PTAL?

[jingyih]

@jingyih I check the code and find that it is hard to monitor whether the member is a learner or not real time.

Can we update the isLearner metrics here:

etcd/etcdserver/server.go

Line 2236 in 819a554

case raftpb.ConfChangeAddNode, raftpb.ConfChangeAddLearnerNode:

Whenever there is an raft entry to add node / add learner node / promote node, and the entry's node ID is the same as local member ID, update learner status.

[WIZARD-CXY]

@jingyih I check the code and find that it is hard to monitor whether the member is a learner or not real time.

Can we update the isLearner metrics here:

etcd/etcdserver/server.go

Line 2236 in 819a554

case raftpb.ConfChangeAddNode, raftpb.ConfChangeAddLearnerNode:

Whenever there is an raft entry to add node / add learner node / promote node, and the entry's node ID is the same as local member ID, update learner status.

good idea!

[WIZARD-CXY]

@jingyih ptal. Changed according to your suggestions and I tested on my local machine. I set up a 3node cluster and add one learner. below is part of merics result. test passed.

➜  ~ curl http://127.0.0.1:2379/metrics 2>&1 |grep learner
# HELP etcd_server_is_learner Whether or not this member is a learner. 1 if is, 0 otherwise.
# TYPE etcd_server_is_learner gauge
etcd_server_is_learner 0
# HELP etcd_server_learner_promote_successes The total number of successful learner promotions while this member is leader.
# TYPE etcd_server_learner_promote_successes counter
etcd_server_learner_promote_successes 0
➜  ~ curl http://127.0.0.1:23790/metrics 2>&1 |grep learner
# HELP etcd_server_is_learner Whether or not this member is a learner. 1 if is, 0 otherwise.
# TYPE etcd_server_is_learner gauge
etcd_server_is_learner 0
# HELP etcd_server_learner_promote_successes The total number of successful learner promotions while this member is leader.
# TYPE etcd_server_learner_promote_successes counter
etcd_server_learner_promote_successes 0
➜  ~ curl http://127.0.0.1:22379/metrics 2>&1 |grep learner 
# HELP etcd_server_is_learner Whether or not this member is a learner. 1 if is, 0 otherwise.
# TYPE etcd_server_is_learner gauge
etcd_server_is_learner 0
# HELP etcd_server_learner_promote_failures The total number of learner promote failures (likely learner not ready) while this member is leader.
# TYPE etcd_server_learner_promote_failures counter
etcd_server_learner_promote_failures{Reason="membership: ID not found"} 1
etcd_server_learner_promote_failures{Reason="membership: can only promote a learner member"} 1
# HELP etcd_server_learner_promote_successes The total number of successful learner promotions while this member is leader.
# TYPE etcd_server_learner_promote_successes counter
etcd_server_learner_promote_successes 1

[jingyih]

lgtm after nits.

[WIZARD-CXY]

@jingyih updated ptal

[jingyih]

lgtm

[jingyih]

Can you send another PR to add the new metrics to the changelog 3.4?
https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.4.md#metrics-monitoring

[WIZARD-CXY]

@jingyih consider it done