MINOR: Adding KRaft Monitoring Related Metrics to docs/ops.html #12679

niket-goel · 2022-09-22T23:30:27Z

This commit adds KRaft monitoring related metrics to the Kafka docs (docs/ops.html).

All metrics contained in the following classes have been added to the commit:

BrokerServerMetrics
QuorumControllerMetrics
KafkaRaftMetrics

KRaft related KIPs mention more metrics, but this commit only adds in the metrics available in the source code at the time of authoring.

niket-goel · 2022-09-22T23:35:07Z

docs/ops.html

+  <tr>
+    <td>Current State</td>
+    <td>The current state of this member; possible values are leader, candidate, voted, follower, unattached.</td>
+    <td>kafka.raft:type=raft-metrics,name=Current-state</td>


Not sure if the type etc is correct here.

Please correct the mbean name:
kafka.server:type=raft-metrics
And below. Thanks.

niket-goel · 2022-09-22T23:36:35Z

docs/ops.html

+  </tr>
+  <tr>
+    <td>Active Controller Count</td>
+    <td></td>


None of the metrics available in QuorumControllerMetrics have a description attached to them. I can write up descriptions for these, but want to make sure that they match up with any literature that we already have. I could not see mentions of these in the KIPs I looked at either. What would be the best course of action here?

niket-goel · 2022-09-22T23:37:13Z

docs/ops.html

+    <td>Metadata Apply Error Count</td>
+    <td>The number of errors encountered by the BrokerMetadataPublisher while applying a new MetadataImage based on the latest MetadataDelta.</td>
+    <td>kafka.server:type=broker-metadata-metrics,name=metadata-apply-error-count</td>
+  </tr>


Assuming this is the complete set of metrics, based on the files I went through. Please point out any files/KIPs that I should look at to add more metrics here.

showuon

@niket-goel , thanks for the PR to add the missing KRaft metrics. I found there are still some metrics missing in this PR. I suggested you run a KRaft kafka in local to get all the metrics it has, then map back to the code/KIP to get the description.

showuon · 2022-09-23T01:21:24Z

docs/ops.html

@@ -1815,6 +1815,162 @@ <h4 class="anchor-heading"><a id="remote_jmx" class="anchor-link"></a><a href="#
      </tr>
  </tbody></table>

+<h5 class="anchor-heading"><a id="kraft_monitoring" class="anchor-link"></a><a href="#kraft_monitoring">KRaft Metrics</a></h5>
+All of the following metrics allow monitoring of the KRaft quourm and 


and [what?]

Hmm.! how did that get deleted. Fixing it. Thanks for catching.

showuon · 2022-09-23T01:25:33Z

docs/ops.html

+    <td>The time in milliseconds to elect a new leader.</td>
+    <td>kafka.raft:type=raft-metrics,name=election-latency</td>


The description of election latency is not clear to me. Maybe:
The time in milliseconds spent on electing a leader.
WDYT?

Also, there is no election-latency metrics, only
election-latency-avg
election-latency-max

I was trying to remain true to the code. I agree that it is not the best description. Let me fix both.

RE: the avg and max I think I made a mistake and picked up the sensor name instead of the metric name. this is tru for another few metrics here. Will fix all.

showuon · 2022-09-23T01:40:40Z

docs/ops.html

+  <tr>
+    <td>Current State</td>
+    <td>The current state of this member; possible values are leader, candidate, voted, follower, unattached.</td>
+    <td>kafka.raft:type=raft-metrics,name=Current-state</td>


Please correct the mbean name:
kafka.server:type=raft-metrics
And below. Thanks.

showuon · 2022-09-23T01:45:12Z

docs/ops.html

+  <tr>
+    <td>Fetch Records</td>
+    <td>The average number of records fetched from the leader of the raft quorum.</td>
+    <td>kafka.raft:type=raft-metrics,name=fetch-records</td>


name: fetch-records-rate
Are your sure it is the avg number?

The description in code says that it is. Let me verify the value and update both of needed.

showuon · 2022-09-23T01:45:43Z

docs/ops.html

+  <tr>
+    <td>Poll Idle Ratio</td>
+    <td>The average fraction of time the client's poll() is idle as opposed to waiting for the user code to process records.</td>
+    <td>kafka.raft:type=raft-metrics,name=poll-idle-ratio</td>


name=poll-idle-ratio-avg

ack! Will fix all metrics coming from this file.

metric description and naming to align with their usage.

niket-goel · 2022-09-23T19:24:45Z

@showuon Thanks for the careful feedback. I have updated the PR with the correct Mbean names and fixed many typos. Please take a look when you have a chance.

* Split the metrics into Broker and Controller Sections * Changed the num-unknown-voter-connections to reflect the KIP

docs/ops.html

Controller

hachikuji · 2022-09-23T23:32:46Z

raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java

-        kafkaRaftMetrics.updateNumUnknownVoterConnections(quorum.remoteVoters().size());
+        // All Raft voters are statically configured and known at startup
+        // so there are no unknown voter connections. Report this metric as 0.
+        kafkaRaftMetrics.updateNumUnknownVoterConnections(0);


Perhaps we should file a jira to follow up with removal of this metric if it is not reporting anything useful.

Yeah! Was planning on filing a JIRA and KIP for its removal. Will do that next week. :)

docs/ops.html

hachikuji

LGTM. I will let @showuon take one more look before we merge this. Thanks for the patch!

showuon

LGTM! Thanks!

showuon · 2022-09-26T06:24:28Z

Failed tests are unrelated:

Build / JDK 11 and Scala 2.13 / kafka.admin.LeaderElectionCommandTest.[1] Type=Raft-Distributed, Name=testPathToJsonFile, MetadataVersion=3.3-IV3, Security=PLAINTEXT

This commit adds KRaft monitoring related metrics to the Kafka docs (docs/ops.html). Reviewers: Jason Gustafson <jason@confluent.io>, Luke Chen <showuon@gmail.com>

showuon · 2022-09-26T06:27:12Z

backport to v3.3

…he#12679) This commit adds KRaft monitoring related metrics to the Kafka docs (docs/ops.html). Reviewers: Jason Gustafson <jason@confluent.io>, Luke Chen <showuon@gmail.com>

…he#12679) (#12) This commit adds KRaft monitoring related metrics to the Kafka docs (docs/ops.html). Reviewers: Jason Gustafson <jason@confluent.io>, Luke Chen <showuon@gmail.com> Co-authored-by: Niket <niket-goel@users.noreply.github.com>

Adding KRaft Monitoring Related Metrics to docs/ops.html

5bed465

niket-goel force-pushed the kraft-ops-monitoring-update branch from 3f48418 to 5bed465 Compare September 22, 2022 23:33

niket-goel commented Sep 22, 2022

View reviewed changes

niket-goel marked this pull request as ready for review September 22, 2022 23:37

showuon reviewed Sep 23, 2022

View reviewed changes

This commit addresses some review feedback and refactors some existing

19cbbdb

metric description and naming to align with their usage.

niket-goel added 2 commits September 23, 2022 14:44

More review comments

5fa7b52

* Split the metrics into Broker and Controller Sections * Changed the num-unknown-voter-connections to reflect the KIP

minor: whitespace fix

51d0e40

hachikuji reviewed Sep 23, 2022

View reviewed changes

docs/ops.html Outdated Show resolved Hide resolved

hachikuji reviewed Sep 23, 2022

View reviewed changes

docs/ops.html Show resolved Hide resolved

Splitting raft metrics to another table and adding the definition of a

de71a6c

Controller

hachikuji reviewed Sep 23, 2022

View reviewed changes

minor rewording

cc104db

hachikuji approved these changes Sep 24, 2022

View reviewed changes

showuon approved these changes Sep 26, 2022

View reviewed changes

showuon changed the title ~~Adding KRaft Monitoring Related Metrics to docs/ops.html~~ MINOR: Adding KRaft Monitoring Related Metrics to docs/ops.html Sep 26, 2022

showuon merged commit eb8f0bd into apache:trunk Sep 26, 2022

niket-goel mentioned this pull request Sep 28, 2022

Update and publish docs for 3.3 apache/kafka-site#447

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MINOR: Adding KRaft Monitoring Related Metrics to docs/ops.html #12679

MINOR: Adding KRaft Monitoring Related Metrics to docs/ops.html #12679

niket-goel commented Sep 22, 2022

niket-goel Sep 22, 2022

showuon Sep 23, 2022

niket-goel Sep 22, 2022

niket-goel Sep 22, 2022

showuon left a comment

showuon Sep 23, 2022

niket-goel Sep 23, 2022

showuon Sep 23, 2022

showuon Sep 23, 2022

niket-goel Sep 23, 2022

showuon Sep 23, 2022

showuon Sep 23, 2022

niket-goel Sep 23, 2022

showuon Sep 23, 2022

niket-goel Sep 23, 2022

niket-goel commented Sep 23, 2022

hachikuji Sep 23, 2022

niket-goel Sep 24, 2022

hachikuji left a comment

showuon left a comment

showuon commented Sep 26, 2022

showuon commented Sep 26, 2022

		<td>The time in milliseconds to elect a new leader.</td>
		<td>kafka.raft:type=raft-metrics,name=election-latency</td>

MINOR: Adding KRaft Monitoring Related Metrics to docs/ops.html #12679

MINOR: Adding KRaft Monitoring Related Metrics to docs/ops.html #12679

Conversation

niket-goel commented Sep 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

showuon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niket-goel commented Sep 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hachikuji left a comment

Choose a reason for hiding this comment

showuon left a comment

Choose a reason for hiding this comment

showuon commented Sep 26, 2022

showuon commented Sep 26, 2022