Reusable integration policies

[nimarezainia]

Describe the enhancement:
Currently the Elastic agents can only belong to single policy. Group of agents in a policy however may have different permutations of applications and data sources to to ingest data from. The 1:1 relationship between agent and the policy it belongs to therefore forces our users to create multiple policies with some having the same integration configuration in multiple policies.

Ideally we could allow a base policy to inherit other "integrations policies" so that they can be used in other base policies. The user then would only need to configure the integration once (smaller management surface for them) and ad that policy to other policies.

Describe a specific use case for the enhancement or feature:

[STRSistemas]

The truth is that before I found this issue, our technical team had also been missing this functionality for months.

It would be a good improvement.

If you look at it from the "agent policy by server type" point of view, I want both my MySQL server types and my Kubernetes-node server types to have the "System" integration, but obviously you don't want/can't have MySQL integration on a Kubernetes node that doesn't have MySQL and vice versa.

If in this situation, having many "server types" we need to make a change in how the "System" integration is used we have to make the same change "N" times for each server type

[sunilemanjee]

This is similar to how policies are built with Splunk UF. Stacking. Glad we are moving in the right direction.

[Hythloday-zero]

+1 for this feature: It would also follow the composable template convention from our Data Streams. IHAC who would benefit greatly from this ( >1k Agents)

[jamesdowell2022]

Very Big UK Bank mentioned something similar last year - building block policies for agent policies, e.g. a basic windows desktop or server policy that they can then add a specific integration to, like we have building block rules in SIEM. Comment from Bank (low priority as they use standalone agent right now - deployed via MDM, in basic form - logs and metrics mostly):

We would also like to get the ability for the policy file to be built from multiple policies rather than everything having to be in one policy, this is due every server having a base policy then on top of that additional integrations would want to be added but only to some servers. We could end up with 100s of policies which would mean at present having to duplicate the base policies every time for the additional policies will lead to mistakes being made and if the base policy needed updating it would have to be done multiple times.

elastic/beats#27442

[nerophon]

We wonder if this is also an opportunity to permit multiple outputs per Fleet-managed Agent. It seems to imply it.

[nerophon]

Further input from an enterprise user:

One thing that this would need is security around the policies, space specific at least and auditing on who has changed a policy.

[nimarezainia]

Further input from an enterprise user:

One thing that this would need is security around the policies, space specific at least and auditing on who has changed a policy.

could you please unicast me more information on this request including who the user is and perhaps we can discuss with them live. thanks

[Oddly]

We are running into the same issue at the moment. Because of how our infrastructure is separated, we will end up with hundreds of policies.

[mbudge]

We are also running into this.

Too many policies when elastic-agent is deployed in many network zones, on-prem, cloud, some collecting application logs, some collecting database logs etc. It get's complicated very quick.

Would also be good to set the system integration metrics data direct to elasticsearch when using logstash for log data.

This will also de-risk the system metrics integration. We found it can cause performance issues on some production servers running under load when using the default settings like 10seconds. We up these to 1minute or 5 minutes for disk/sockets to reduce load on all hosts.

[user-987654321]

+1