Update the agent integration dashboard with additional filebeat and metricbeat metrics

[mukeshelastic]

Describe the enhancement:
We added the awesome elastic agent integration in 7.11 that shows agent metrics and filebeat and metricbeat system metrics. There are other meaningful filebeat and metricbeat metrics that we gather but are not showing it on dashboard today. These metrics are displayed in stack monitoring UI and are described in our documentation

Acceptance criteria:
After this issue is resolved, users will be able to see all the filebeat and metricbeat metrics that we display in stack monitoring UI.

@ravikesarwani I do not specific metric names other than the ones in documentation. If there are additional metrics not shown but we need to show in dashboard, can you please add them in this issue?

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ph]

see screenshot below for a more update to date UI

@ravikesarwani @mukeshelastic Just to clarify you want to display all the above metrics in the dashboard?

[ph]

@nchaulet since you did the initial implementation, might be easier if you do the followup, Could you give me an idea of the level of effort to add the following graph and the other information in the dashboard?

[ravikesarwani]

Metricbeat monitoring shows the following graphs:

[ravikesarwani]

@ph The image you pasted may have few things missin. Here's an image for filebeat that shows the identical data as metricbeat.

[ph]

thanks @ravikesarwani, I've removed my screenshot to make sure there is no misunderstanding, the picture was coming from the above documentation link

[ravikesarwani]

One area that we got multiple requests is around showing cgroup information for CPU and memory. This is important to show information correctly in container environment. Beats made the change to make that data available and stack monitoring will make the change in 7.1.2 (elastic/kibana#79050) for beats and APM server monitoring.
For Agent metrics dashboard we need to show this data as well to properly account for container environments.
For Elasticsearch we show the following graph:

[nchaulet]

We have the data for the cgroup so we can probably add that to the existing dashboard.

we have the metrics for to add Events Rates/Fail Rates/ Throughput/Output Errors but how do we want to present that considering our dashboard is for multiple process

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ravikesarwani]

@nchaulet We discussed and used the stacked area chart to begin with. Here we are trying to primarily show how the whole changes through time and also give a very rough sense of how the parts compare to one another in their contributions to the whole at particular points of time. This graph does have drawback as understanding each part and how it changes over time in not very clear (other than the bottom component).

Overall I think we need different aspects of the time-series/part-to-whole story we want to tell will require different graphical approaches and cannot be combined in a single graph efficiently. We can potentially use 2 charts:

• One shows the Agent as a whole with a line chart.
• The second chart shows each component (Agent, filebeat, metricbeat...) as its own line with a unique color.
  With the two charts (top/bottom) the user can visualize both data point efficiently.

This of course adds a lot more visualizations to the dashboard.

My first take would be to add all the other values in the same way as stacked area chart. Once you have added more things let me know and I can take a look. We can also discuss this with someone with a more SRE background and get their opinion as well.

Let me know you thoughts?

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)