[elastic_agent] Agent Metrics Dashboard improvements

[leehinman]

Proposed commit message

The visualizations in the "[Elastic Agent] Agent Metrics" Dashboard have been improved. Specifically:

For the following visualizations the breakdown was changed from beat.type to agent.name+component.id. This was necessary because it was grouping data together from different sources which was reporting incorrect values. agent.name+component.id is correct because this maps the data to the unique process id that generated the data.

Also, the interval was changed from auto to minute. This is necessary because the data only comes in at 1 minute intervals. By default the visualization was grabbing smaller intervals which was breaking the visualization.

• [Elastic Agent] Total events rate /s
• [Elastic Agent] Output write throughput
• [Elastic Agent] Output write errors
• [Elastic Agent] Events acknowledged rate /s
• [Elastic Agent] Output write batches
• [Elastic Agent] Output batch size

The following visualizations were added.

• [Elastic Agent] Events dropped rate /s
• [Elastic Agent] Events duplicate rate /s
• [Elastic Agent] Events failed rate /s
• [Elastic Agent] Events TooMany rate /s

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• I used https://github.com/elastic/mock-es to generate erros that would show up in dropped, duplicate, failed and TooMany visualization
• I used https://github.com/leehinman/spigot to generate events at predictable rates and verified those rates were reported by the visualizations

How to test this PR locally

1. Install elastic-agent
2. Look at [Elastic Agent] Agent Metrics Dashboard

Related issues

• Closes Agents Metrics dashboards / visualizations are incorrect since the tsds was introduced with the new counter type #9724

Screenshots

Before

After

All Visualization

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[fearful-symmetry]

@leehinman do you want to merge from upstream? The conflicts are probably me. Hoping I'll actually have time to get to this tomorrow.

[fearful-symmetry]

So, this is probably a me problem, but I can't get the CPU usage dashboard to show anything? This is just the "default" elastic-package deployment (elastic-package stack up) with a system integration installed and agent metrics enabled to get more data. Can you verify this is working?

It looks like the visualization is aggregating from system.process.cpu.total.value and elastic-agent.process, and there are documents matching with those fields.

The query response shows that we have matching documents, but the aggregation isn't there.

{
  "rawResponse": {
    "took": 7,
    "timed_out": false,
    "_shards": {
      "total": 15,
      "successful": 15,
      "skipped": 14,
      "failed": 0
    },
    "hits": {
      "total": 75,
      "max_score": null,
      "hits": []
    }
  }
}

Again, might just me being dumb, but it seems odd, since the Memory Usage and Open Handles metrics are working fine?

[leehinman]

Can you verify this is working?

I didn't touch the "CPU Usage" chart, but I figured out what is wrong. It is the fact that the data view is set to "auto" and we only get events every minute. So if auto picks buckets that are smaller than one minute (which is what happens with default 15 min time picker) Some of the buckets have no data, and things like derivations don't work. :-(

fix coming soon.

[leehinman]

@fearful-symmetry try it now

[jlind23]

@leehinman @pierrehilbert is this good to be merged?

[leehinman]

@leehinman @pierrehilbert is this good to be merged?

Marius is out on paternity leave, I added Joe Reuter as reviewer, I'd like someone with a bit more Dashboard experience to review just to make sure I'm not missing something.

[flash1293]

Thanks for this. Some comments.

I ran this on some sample data and the first chart errors out for me:

Didn't dig deeper on why that's happening, but it seems to be a TSVB vis, is there a reason for this? By my understanding we shouldn't use TSVB unless absolutely required.

In terms of the charts - all the time series charts are pretty dense and the y axis label doesn't really add much as the title is explaining it quite well already - I think we can omit them:

For the x axis, it's pretty much the same - IMHO we can just remove the label here, the "per minute" is more confusing than helpful as we normalize the value per second anyway:

[leehinman]

Thanks for this. Some comments.

I ran this on some sample data and the first chart errors out for me:

Didn't dig deeper on why that's happening, but it seems to be a TSVB vis, is there a reason for this? By my understanding we shouldn't use TSVB unless absolutely required.

Looks like this was done when Dashboards were first made. I'm going to leave it as TSVB until Marius bets back and we can ask him about it, since he did the original work.

I think the error is "too much data", and I'm guessing that is because I set the interval to "1m", I switched it to ">=1m", lets see if that fixes it. If we go less than 1m, we have a different problem.

In terms of the charts - all the time series charts are pretty dense and the y axis label doesn't really add much as the title is explaining it quite well already - I think we can omit them:

I didn't find a way to "remove" the text, I can change it to " ", so nothing shows up, but I can't seem to remove it totally and reclaim the space. Do you know of a better way?

For the x axis, it's pretty much the same - IMHO we can just remove the label here, the "per minute" is more confusing than helpful as we normalize the value per second anyway:

These are automatically added, so I could remove the "@timestamp" part of "@timestamp per minute", but I didn't find a way to remove the "per minute" part. Any ideas?

[leehinman]

Updated dashboards look like:

[flash1293]

I think the error is "too much data", and I'm guessing that is because I set the interval to "1m", I switched it to ">=1m", lets see if that fixes it. If we go less than 1m, we have a different problem.

That should work 👍

I didn't find a way to "remove" the text, I can change it to " ", so nothing shows up, but I can't seem to remove it totally and reclaim the space. Do you know of a better way?

You can remove the axis titles in the axis menu like this:

[leehinman]

@flash1293 Thanks for the help on the axis titles. 2 of the the panels are different and don't have those options, but I got it fixed on the rest. screenshot below.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 4628615

History

• 💚 Build #12536 succeeded a5492d8
• 💚 Build #12332 succeeded db7f9e6
• 💚 Build #12272 succeeded 5ff8e0c
• 💚 Build #12115 succeeded 17ca98b

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[flash1293]

yeah, for tsvb you can't remove it I think - LGTM from the dashboard side based on the screenshots

[elasticmachine]

Package elastic_agent - 1.20.0 containing this change is available at https://epr.elastic.co/search?package=elastic_agent