[Elastic Agent] How to monitor inputs

[ruflin]

In Fleet a user deploys and Integration policy as part of an Agent policy. From an Agent perspective, an integration policy is just an array of inputs. The user needs to know if an Integration policy is working or not. For an Agent to report if an integration works or not, the Agent needs to know about the status of each input. As the Agent is mostly a supervisor and the inputs are run by the processes, this data needs to be reported by the processes.

This issue is to discuss the following points:

• What are the metrics that each input should report
• How is each input going to report this metrics

Proposal on reporting path

To take a first stab on how this could work, here a proposal on the part on how the metrics are reported. Today metrics and logs are collected from a separate Metricbeat / Filebeat instance. I would not change this for now but introduce an additional path for the input monitoring. Each process has a standardised way on how to report inputs metrics to the Elastic Agent through GRPC. Then the Agent decides on how all these metrics are shipped / reported to Elasticsearch / Fleet.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

@belimawr will start working on it.
@kvch how does it match with the input manager proposal you wrote?

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!