Updated Logstash Single Pipeline View to display batch size and event count metrics

[andsel]

Proposed commit message

Update a couple of Logstash metrics dashboard to display batch size and event count. This feature is available since Logstash 9.2

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

The test is based on a running Logstash instance monitored by an ElasticAgent which push metric data to be processed by the integration.

Requirements: Docker must be running on test host.

1. Install elastic-package

   1. Download from https://github.com/elastic/elastic-package/releases/tag/v0.117.1

   2. Add permission to the file with

      xattr -r -d com.apple.quarantine elastic-package

   3. Run it

      ./elastic-package

2. Bring up stack and build/install the integration (from integration/packages/logstash of the Elastic Integration local clone)

   1. Launch the stack

      /path/to/elastic-package stack up -v

   2. Build the package locally

      /path/to/elastic-package build

   3. Install into the package registry running in the Docker, so it can be served to Fleet

      /path/to/elastic-package install

   4. Verify the integration is installed, check the listing: https://localhost:5601/app/integrations/installed?currentPage=1

3. Now install and configure a local Logstash that generates some metrics, this is monitored by a local agent that will be installed in next step

   1. Download Logstash >= 8.2.0 from https://www.elastic.co/downloads/logstash

   2. Once unpacked edit the config/logstash.yml to have the following settings:

      pipeline.batch.metrics.sampling_mode: "full"
      monitoring.enabled: false # this is already the default

   3. Run Logstash with a pipeline, rememeber to check which HTTP API port is bound, usually 9600 but since the Docker compose already setup a Logstash instance in the container, it could be 9601 or so:

      bin/logstash -e "input{ tcp {port => 3333} } output{ sink{} }"
      

   4. Generate some flow with a simple script (requires JDK and JBang installed locally, use sdkman to do that). Use the gist https://gist.github.com/andsel/d3b372b90bd66e0db98a6acc1ac32c80 and run with:

      ./TrafficSimulator.java -p 3333

4. Add Logstash integration and create new policy and enroll a local agent that monitor the launched Logstash:

   1. In Fleet go to the Logstash integration and press Add Logstash

   2. Create a new policy and enroll a new agent

   3. Follow the instructions to download the agent and run it, but to the proposed command line remember to add: --develop --insecure. Add --develop to be able run side by side the existing Agent, and --insecure to avoid x509 certificate verification. As example should be something like:

      ./elastic-agent install --url=https://localhost:8220 --enrollment-token=<PROVIDED TOKEN> --develop --insecure

   4. In the configuration of the policy set the port where local Logstash bound (step 3.iii)

   5. It may require to update your ´/etc/hosts´ file to avoid some error in Fleet UI like "Error get fleet-server" or something related to reaching elasticsearch, in case add the following to your hosts:

      127.0.0.1       fleet-server
      127.0.0.1       elasticsearch #I'm not sure of this!
      

5. Verify the dashboard [Metrics Logstash] Logstash Single Pipeline View

   1. Scroll down the dashboard there are 2 new graphs that display the batch size and event count.

Related issues

• Closes Add Batch size metrics to the Logstash Integration logstash#18440

Screenshots

[robbavey]

• Current positioning of graphs is difficult to discover, at the bottom of the page in the "pipelines details" page, and not aligned with the other graphs.
  • I would suggest moving the graphs to the top of the page to be aligned with the other pipeline-level graphs
• Consider also adding these graphs to the pipeline overview page, in a similar position.
• What would be the effort of adding actual batch size metrics to the pipeline table?

* I'm not sure what the "average (last 1 minute) axis" adds - from some quick testing it essentially seemed to be the same as "current", but shifted over a minute. Do we actually need it? * The information field might be better used to note that this graph requires the version xxx and above of Logstash

[robbavey]

Also, to fix the build, I suspect you are going to need to change the Logstash pipeline definitions to conform to the 9.x ssl standardized config names

[andsel]

Hi @robbavey , I've updated the two dashboards as you asked:

Pipeline Overview

Logstash single pipeline view

I've also update the LS pipeline definitions to respect the 9.x deprecations in ES output, but still the CI is red and can't figure out how to discover which is the culprit.

[robbavey]

@andsel - you can view the Logstash logs from the container of the buildkite box by following these instructions:

https://docs.elastic.dev/ingest-dev-docs/elastic-packages/ecosystem-ci-pipelines#private-logs

The root cause:

logstash-1  | Using bundled JDK: /usr/share/logstash/jdk
logstash-1  | Warning: no jvm.options file found.
logstash-1  | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash-1  | 2026-02-18 09:34:30,113 main ERROR Unable to locate appender "pipeline_routing_appender" for logger config "root"
logstash-1  | [2026-02-18T09:34:30,146][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<RuntimeError: Logstash cannot be run as superuser.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/runner.rb:440:in `running_as_superuser'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:266:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.3.3/lib/clamp/command.rb:66:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:256:in `run'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.3.3/lib/clamp/command.rb:140:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:89:in `<main>'"]}
logstash-1  | [2026-02-18T09:34:30,190][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
logstash-1  | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash-1  | 	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:924) ~[jruby.jar:?]
logstash-1  | 	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:883) ~[jruby.jar:?]
logstash-1  | 	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

Updating the logstash.yml to add allow_superuser:true is the short term fix, but there should be an issue to not require logstash to run this way in the test runner.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[mashhurs]

CEL scripts and dashboards are overall LGTM!
I have asked some questions (why super user required, taking out gauge) to clarify and I do think we don't necessarily need a SECTION (we used sections for plugin types) and introduced metrics are belong to pipeline (common like worker utlization, etc..)

And please also update the migration version, it looks like you had a local snapshot.

Thank you for work!

[andsel]

I do think we don't necessarily need a SECTION

This is a layout need, without that there is an hole from the above graphs, and we can't have these 2 graphs on the same line:

[mashhurs]

I do think we don't necessarily need a SECTION

This is a layout need, without that there is an hole from the above graphs, and we can't have these 2 graphs on the same line:

I was thinking just remove this (markdown?) and re-arrange the batch visualizations, something like this:

[andsel]

@mashhurs

I was thinking just remove this (markdown?) and re-arrange the batch visualizations, something like this:

I think that that layout is less intuitive, mix batch structure graphs (those 2 vertical) with the one about evt/s. In theory with the percentiles we add even more here.

[mashhurs]

@mashhurs

I was thinking just remove this (markdown?) and re-arrange the batch visualizations, something like this:

I think that that layout is less intuitive, mix batch structure graphs (those 2 vertical) with the one about evt/s. In theory with the percentiles we add even more here.

Yep, we can address this easily if it brings any confusions.

Review already done by @mashhurs

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 281f871

History

• 💚 Build #39297 succeeded 34a36dc
• 💚 Build #39222 succeeded 67c722c
• 💚 Build #39113 succeeded 69ac87d
• 💚 Build #38934 succeeded bc16ad9
• 💚 Build #38838 succeeded 1895063
• 💔 Build #38831 failed 44336ba

cc @andsel @robbavey

[elastic-vault-github-plugin-prod]

Package logstash - 2.9.0 containing this change is available at https://epr.elastic.co/package/logstash/2.9.0/