Skip to content

Conversation

@mashhurs
Copy link
Contributor

@mashhurs mashhurs commented Sep 3, 2025

Proposed commit message

Save current and peak connection metrics of the Logstash elastic_agent/beats input plugin and visualize its metrics on dashboard.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Check dashboard metrics with team members.

How to test this PR locally

Related issues

Screenshots

screencapture-localhost-5601-app-dashboards-2025-09-04-10_41_23

@mashhurs mashhurs self-assigned this Sep 3, 2025
Copy link
Contributor Author

@mashhurs mashhurs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@robbavey I have added metrics on the dashboard which are available: [current_connections, peak_connections, events.out, queue_push_duration].
I have also attached the visualization view. WDYT?

"counter_map": has(body.pipelines[pipeline_name].vertices) ?
body.pipelines[pipeline_name].vertices.map(vertex, has(vertex.long_counters), vertex.long_counters.map(counter, {
body.pipelines[pipeline_name].vertices.map(vertex, has(vertex.long_counters) || has(vertex.double_gauges),
((has(vertex.long_counters) ? vertex.long_counters : []) + (has(vertex.double_gauges) ? vertex.double_gauges : [])).map(counter, {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review note: peak and current connections are gauge metrics and comes through _node/stats?vertices=true API response -> vertices -> double_gauges
See: elastic/logstash#18082

@mashhurs mashhurs marked this pull request as ready for review September 3, 2025 18:34
@mashhurs mashhurs requested a review from a team as a code owner September 3, 2025 18:34
@robbavey
Copy link
Member

robbavey commented Sep 3, 2025

How would current_connections and peak_connections on the same graph look? I seem to recall that we do something like that for file_descriptors on the extended node dashboard?

@andrewkroh andrewkroh added Integration:logstash Logstash dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Sep 3, 2025
@mashhurs
Copy link
Contributor Author

mashhurs commented Sep 3, 2025

How would current_connections and peak_connections on the same graph look? I seem to recall that we do something like that for file_descriptors on the extended node dashboard?

Yup, they together also fine I think, It becomes looks like this:
screencapture-localhost-5601-app-dashboards-2025-09-03-14_01_29

@mashhurs mashhurs requested a review from robbavey September 4, 2025 18:25
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit on naming, and a question on changelog/release strategy

# newer versions go on top
- version: "2.8.0"
changes:
- description: Adds elastic_agent or beats input plugin current and peak connections dashboard.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this won't work until elastic/logstash#18082 and friends are in released versions of Logstash, should we a) hold off on publishing until a compatible version(s) of Logstash is available (or at least imminent - we can release note this in the appropriate versions of Logstash), and b) note which versions of Logstash are required to enable this plugin.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, on Sept. 16, I will add a note here about LS versions: 8.18.7, 8.19.4, 9.0.7, 9.1.4

@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Sep 9, 2025

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but should be rebased over #14457 once that is merged

@mashhurs mashhurs force-pushed the support-gauge-metrics-in-ls-integration branch from f6610b3 to 7bd74d9 Compare September 17, 2025 15:06
@elasticmachine
Copy link

💚 Build Succeeded

History

cc @mashhurs

@elastic-sonarqube
Copy link

@mashhurs mashhurs merged commit 399ac8f into elastic:main Sep 18, 2025
9 checks passed
@mashhurs mashhurs deleted the support-gauge-metrics-in-ls-integration branch September 18, 2025 17:20
tehbooom pushed a commit to tehbooom/integrations that referenced this pull request Nov 19, 2025
…astic-agent input plugin. (elastic#15148)

* Saves current and peak connections metrics of elastic-agent input plugin and visualizes on dashboard.

* Update the type migration version which was snapshot before.

* Unify current and peak connections graphs into single view.

* Rename y-axis title.

* Rename the dashboard title and panel.

* Specify Logstash versions which expose these metrics.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. >enhancement Integration:logstash Logstash

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants