Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Confluent Platform Intergration - Correct Mbean regex stream thread metric #17490

Merged
merged 7 commits into from
May 29, 2024
+103 −1
Merged

fix: Confluent Platform Intergration - Correct Mbean regex stream thread metric #17490

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5f318c3
fix: stream client metric to thread metric
tlsdnwn55 Apr 30, 2024
e0bafc0
Add: change log
tlsdnwn55 May 1, 2024
0a2c164
add better change log
tlsdnwn55 May 4, 2024
8aee7a4
Merge branch 'master' into fix/conflunent-platform-intergration
tlsdnwn55 May 7, 2024
d07f286
add: legacy stream theard metric for backwards compatibility with at…
May 9, 2024
03db929
Merge branch 'master' into fix/conflunent-platform-intergration
tlsdnwn55 May 23, 2024
4bc637c
fix: appropriate change log
tlsdnwn55 May 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions confluent_platform/changelog.d/17490.fixed
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@

Fix correct Mbean regex stream thread metric(after version 5.5.0 of confluent platform). Previously, because Bean Regex was set to Client metrics, Stream Thread Metrics were not collected. also added one useful attribute for debugging Stream Thread performance.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Fix correct Mbean regex stream thread metric(after version 5.5.0 of confluent platform). Previously, because Bean Regex was set to Client metrics, Stream Thread Metrics were not collected. also added one useful attribute for debugging Stream Thread performance.
Add a Bean regex for stream thread metrics for Confluent version >=5.5.0. Previously, Stream Thread Metrics were not collected. Also added one useful attribute for debugging stream thread performance.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks I think your suggestion is more appropriate.

102 changes: 101 additions & 1 deletion confluent_platform/datadog_checks/confluent_platform/data/metrics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1600,7 +1600,107 @@ jmx_metrics:


# Components Metrics:
# - Confluent Streams Thread: https://docs.confluent.io/current/streams/monitoring.html#built-in-metrics
# - Confluent Streams Thread: https://docs.confluent.io/current/streams/monitoring.html#thread-metrics
- include:
domain: kafka.streams
bean_regex: kafka\.streams:type=stream-thread-metrics,thread-id=.*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add this entry as new one for this bean regex instead of overwriting the current one? If we overwrite this, it might break some backwards compatibility with at least version 5 of confluent platform:
https://docs.confluent.io/legacy/platform/5.0.0/streams/monitoring.html#:~:text=MBean%3A%20kafka.streams%3Atype%3Dstream%2Dmetrics%2Cthread.client%2Did%3D%5BthreadId%5D

Moving this to a new entry will ensure both legacy and new versions to submit this metric.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@steveny91 Thank you your advice.
Metrics have been added for compatibility up to version 5.4.10.
Please confirm.

attribute:
blocked-time-ns-total:
# The total time the Kafka Streams thread spent blocked on Kafka since it was started, in nanoseconds (ns).
alias: confluent.$domain.stream.$attribute
metric_type: gauge
commit-latency-avg:
# The average value of commit-latency.
alias: confluent.$domain.stream.$attribute
metric_type: gauge
commit-latency-max:
# The maximum value of commit-latency.
alias: confluent.$domain.stream.$attribute
metric_type: gauge
commit-rate:
# The average per-second number of commit calls
alias: confluent.$domain.stream.$attribute
metric_type: gauge
commit-total:
# The total number of commit calls
alias: confluent.$domain.stream.$attribute
metric_type: gauge
poll-latency-avg:
# The average value of poll-latency.
alias: confluent.$domain.stream.$attribute
metric_type: gauge
poll-latency-max:
# The maximum value of poll-latency.
alias: confluent.$domain.stream.$attribute
metric_type: gauge
poll-rate:
# The average per-second number of poll calls
alias: confluent.$domain.stream.$attribute
metric_type: gauge
poll-total:
# The total number of poll calls
alias: confluent.$domain.stream.$attribute
metric_type: gauge
process-latency-avg:
# The average value of process-latency.
alias: confluent.$domain.stream.$attribute
metric_type: gauge
process-latency-max:
# The maximum value of process-latency.
alias: confluent.$domain.stream.$attribute
metric_type: gauge
process-rate:
# The average per-second number of process calls
alias: confluent.$domain.stream.$attribute
metric_type: gauge
process-total:
# The total number of process calls
alias: confluent.$domain.stream.$attribute
metric_type: gauge
punctuate-latency-avg:
# The average value of punctuate-latency.
alias: confluent.$domain.stream.$attribute
metric_type: gauge
punctuate-latency-max:
# The maximum value of punctuate-latency.
alias: confluent.$domain.stream.$attribute
metric_type: gauge
punctuate-rate:
# The average per-second number of punctuate calls
alias: confluent.$domain.stream.$attribute
metric_type: gauge
punctuate-total:
# The total number of punctuate calls
alias: confluent.$domain.stream.$attribute
metric_type: gauge
skipped-records-rate:
# The average per-second number of skipped records
alias: confluent.$domain.stream.$attribute
metric_type: gauge
skipped-records-total:
# The total number of skipped records
alias: confluent.$domain.stream.$attribute
metric_type: gauge
task-closed-rate:
# The average per-second number of closed tasks
alias: confluent.$domain.stream.$attribute
metric_type: gauge
task-closed-total:
# The total number of closed tasks
alias: confluent.$domain.stream.$attribute
metric_type: gauge
task-created-rate:
# The average per-second number of newly created tasks
alias: confluent.$domain.stream.$attribute
metric_type: gauge
task-created-total:
# The total number of newly created tasks
alias: confluent.$domain.stream.$attribute
metric_type: gauge

# Components Metrics:
# - Confluent Streams Thread: https://docs.confluent.io/legacy/platform/5.4.10/streams/monitoring.html#built-in-metrics
# For backwards compatibility with at least version 5.4.10 of confluent platform
- include:
domain: kafka.streams
bean_regex: kafka\.streams:type=stream-metrics,client-id=.*
Expand Down