[Elastic Agent] Add support for dynamics inputs

[ph]

In beats today we have a feature called Autodiscovery that allows you to generate inputs based on labels or other information coming from a provider like the k8s, docker stream. We want to add the same functionality in the Elastic Agent. We are calling that new feature Dynamic input.

As an example, we would create a dynamic input template like this. And the system would run through the data it has to define if a template should be used or not.

inputs:
- type: log
  dataset.type: logs
  streams:
    - paths: /var/log/${host.platform}.log
      dataset.name: foo
  condition: ${host.platform} == "windows"

Use cases:

• Do an input need to run on this OS (aka syslog should not work on Windows).
• Do an input need to run based on the environment.
• Do an input need to run based on data from a provider (k8s hints)
• Allow defining a cross platform integration that will use journald on Linux but winlogbeat on Windows

Tasks

Phase 1: Elastic Agent

• Write a proposal for conditionals language previous discussion at [Elastic Agent] Support EQL condition on an input #20784
• Allow Elastic-Agent to evaluate templates based on defined conditions. [Elastic Agent] Support EQL condition on an input #20784
• Define concepts of input templates and provider in Elastic Agent. [Elastic Agent] Support variable substitution from providers #20781
• System Provider: Allow to retrieve information about the current host. [Elastic Agent] Support variable substitution from providers #20781

Phase 2 (beyond 7.10)

• Debugging Tools: Add ability to debug a provider easily. [Elastic Agent] Add ability to debug a composable provider elastic-agent#123
• Add Keystore support in the Elastic-Agent [Agent] Add support for keystore #14454
• Refactor autodiscovery/docker [Elastic Agent] Add docker composable dynamic provider. #20842
• Refactor autodiscovery/kuberbetes Refactor autodiscover to make it reusable in the Agent #19271

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

cc @andresrc @exekias @ruflin @roncohen @urso

[exekias]

I've created #19271 to discuss the changes in current autodiscover

[ph]

@ruflin @blakerouse One thing that I think our idea doesn't cover is a default case. Lets assume that an integration ships with 3 inputs.

- type: A
    conditions: 
      - {{os.platform}} == windows

- type: B
    conditions: 
      - {{journald_detected}} == true

- type: C
    conditions: 
      - Run only is A/B is not enabled.

The use case would be use the windows events, journald if it exist, otherwise fallback to file on disk? Do we need to support that kind of condition or we assume that everything is a single "unit" with only his own view on the world? The later really simplify any use case.

[ruflin]

I would skip this default use case for now. For these cases != can be used. For the above example, C would be

- type: C
    conditions: 
      - {{os.platform}} != windows
      - {{journald_detected}} != true

[ph]

@ruflin Yes this is what I thought, just wanted to make sure we were ok with possible limitations. Also, in some we could have noop inputs. (IE the file do not exist)

[ph]

Closing this, added the remaining items in the project board.