Update Logstash single pipeline dashboard with batch byte_size metrics for p50 p90 max

[andsel]

Proposed commit message

Update the cel script to store p50 p90 and max byte size metrics of the batch for the last minute.
Switch to use the average last one minute metric instead of the lifetime for both batch's byte size and value to be coherent with the other graphed percentiles.
Update the Single Pipeline View to include these new values.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

The test is based on a running Logstash instance monitored by an ElasticAgent which push metric data to be processed by the integration.

Requirements: Docker must be running on test host.

1. Install elastic-package

   1. Download from https://github.com/elastic/elastic-package/releases/tag/v0.117.1

   2. Add permission to the file with

      xattr -r -d com.apple.quarantine elastic-package

   3. Run it

      ./elastic-package

2. Bring up stack and build/install the integration (from integration/packages/logstash of the Elastic Integration local clone)

   1. Launch the stack

      /path/to/elastic-package stack up -v

   2. Build the package locally

      /path/to/elastic-package build

   3. Install into the package registry running in the Docker, so it can be served to Fleet

      /path/to/elastic-package install

   4. Verify the integration is installed, check the listing: https://localhost:5601/app/integrations/installed?currentPage=1

3. Now install and configure a local Logstash that generates some metrics, this is monitored by a local agent that will be installed in next step

   1. Download Logstash >= 8.2.0 from https://www.elastic.co/downloads/logstash

   2. Once unpacked edit the config/logstash.yml to have the following settings:

      pipeline.batch.metrics.sampling_mode: "full"
      monitoring.enabled: false # this is already the default

   3. Run Logstash with a pipeline, rememeber to check which HTTP API port is bound, usually 9600 but since the Docker compose already setup a Logstash instance in the container, it could be 9601 or so:

      bin/logstash -e "input{ tcp {port => 3333} } output{ sink{} }"
      

   4. Generate some flow with a simple script (requires JDK and JBang installed locally, use sdkman to do that). Use the traffic generator script at https://github.com/andsel/traffic_simulator and run with:

      ./TrafficSimulator.java -p 3333 -e continuous -w exponential

      If JBang is not able to resolve dependencies, check your ~/.m2/settings.xml and eventually remove it.

4. Add Logstash integration and create new policy and enroll a local agent that monitor the launched Logstash:

   1. In Fleet go to the Logstash integration and press Add Logstash

   2. Create a new policy and enroll a new agent

   3. Follow the instructions to download the agent and run it, but to the proposed command line remember to add: --develop --insecure. Add --develop to be able run side by side the existing Agent, and --insecure to avoid x509 certificate verification. As example should be something like:

      ./elastic-agent install --url=https://localhost:8220 --enrollment-token=<PROVIDED TOKEN> --develop --insecure

   4. In the configuration of the policy set the port where local Logstash bound (step 3.iii)

   5. It may require to update your ´/etc/hosts´ file to avoid some error in Fleet UI like "Error get fleet-server" or something related to reaching elasticsearch, in case add the following to your hosts:

      127.0.0.1       fleet-server
      127.0.0.1       elasticsearch #I'm not sure of this!
      

5. Verify the dashboard [Metrics Logstash] Logstash Single Pipeline View

   1. Scroll down the dashboard there are 2 new graphs that display the batch size and event count.

Related issues

• Relates Updated Logstash Single Pipeline View to display batch size and event count metrics #17009
• Closes Update the Elastic integration to present p50 p90 and max batch byte size metrics logstash#18972

Screenshots

[github-actions]

✅ Vale Linting Results

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[robbavey]

Couple of concerns about presentation:

I'm not sure about having multiple areas, and why p50 and p90 are represented by areas, but max is not. WDYT about only p50 being represented by area, or nothing?

average lifetime is difficult to read on this graph too

[andsel]

I imagined p50 an p90 as areas because those make immediately visible the where the current and max are respect to the range that should contains the majority of the measures.
I tried to add also max as an area, but the graph became too much cluttered, because max is an upper limit, I think it's fine to be a line.

If we keep all percentiles plus current and average as lines, the graph it's hard to understand, you have to consider 5 lines and understanding the relationship it's not easy.
The areas permit an immediate separation between percentiles and current, average values.
Let me know if you think that we are presenting too much information in a single graph. I think that the idea of the graph is to provide how current and average relates to percentiles in a glipse.

[robbavey]

@perk @jsvd Do you have any thoughts here?

[perk]

What if we've made the shaded area between p50 and p90, with current and average as lines in between them and max as a line above the shaded area?

But that isn't 100% bulletproof. It feels like we could split this into two separate charts: one for Current vs. Average, and another strictly for Percentiles (p50, p90, Max).

[andsel]

@perk so your suggestion is to keep as it is but only change the shaded areas, maintaining the shading only for the strip between p50 and p90. Am I right?

In that case I don't know hot to do it, I think we need to speak to someone in Kibana hot to that (who?).

It feels like we could split this into two separate charts: one for Current vs. Average, and another strictly for Percentiles (p50, p90, Max).

It could be, but at that point I think it became difficult to correlate current with the percentile measures (and max). Do you think that there is a benefit, in terms of better understanding, being able to correlate percetniles and current ?

[perk]

suggestion is to keep as it is but only change the shaded areas, maintaining the shading only for the strip between p50 and p90. Am I right?

Yes, I think that would be more clear than the current dashboard. But I'm not sure it's feasible either. Is it?

Do you think that there is a benefit, in terms of better understanding, being able to correlate percetniles and current?

Let me know if you think that we are presenting too much information in a single graph.

Fair question. I think that ability to correlate on a single dashboard is a good thing in general.

[andsel]

Playing with color, stacking and custom formula I was able to draw colored stripe that reppresent the p50-p90, as in the following image:

@perk I've one doubt: how effective is to graph the "average lifetime" value for both event count and byte value. If there is an increase or decrease, the time that metrics take to reach a "stable value" is proportional to the time the LS process was running. For example, suppose it was running for some days with pretty steady flow, that originated a an average of 100KB, if there is an increase to 1MB , the average lifetime takes sometime to get close to the 1MB value. I would suggest to use the average last 1 minute, so that all data series reflect the same temporal window.

Let me know what you think.

[perk]

If we call the lifetime average a lifetime one we should calculate it as such.

That being said :) It doesn't have to be lifetime, it could be just average.

In general I agree time windows are mixed here.

Maybe we should have all the values calculated using the sampling rate?
Eg. on your last image you have @timestamp per 30 seconds - in that case shouldn't all the values be calculated using the 30 seconds window? Where does the last minute requirement comes from?

[andsel]

The last minute is an already aggregated and compute datapoint on Logstash side, it comes from flow metric which is used to track batch size values: https://www.elastic.co/docs/api/doc/logstash/operation/operation-nodestatspipelines.

Those values are not computed on Kiabana side but just used as they are.

[perk]

The last minute is an already aggregated and compute datapoint on Logstash side, it comes from flow metric

Thanks for clarification!

I would suggest to use the average last 1 minute, so that all data series reflect the same temporal window.

That makes sense to me now :)

[estolfo]

I saw the same dashboard results as previous comments:

[robbavey]

@andsel Running against 9.4.0, I see a couple of issues:

The data is available for p50, but the visualization is not showing up, maybe due to color choice?

[andsel]

@robbavey
First issue, was my fault, missed to press "save" in Kibana before exporting, now it's ok.

About your second question. That's related to the way the p50-p90 stripe is created. It's obtained as a couple of stacked areas metrics. As first step a p90.last_1_minute - p50.last_1_minute vertical axis is created, with orange color; then the p50 drawn as a "transparent area" with stacked p90-p50 is drawn. If you switch to dark theme you can see it (top right click on the "E" > "Edit profile" > "color mode".
This choice was made to be able to draw that p50-p90 coloured stripe.

[robbavey]

@andsel - Makes sense, the only other thing I would suggest is that the "average last minute" value is also somewhat hard to see in light mode, maybe something darker would help

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f3d9378

History

• 💚 Build #42646 succeeded 5d03615
• 💚 Build #42356 succeeded c94c0f2
• 💚 Build #42351 succeeded 22912b1
• 💚 Build #41524 succeeded 2377e3d
• 💚 Build #41163 succeeded 240db5e

cc @andsel

[andsel]

@robbavey this is the graph with a little darker color for "average last. minute"

[robbavey]

LGTM