[Elastic Agent] No ES output validation can lead to stopped sub processes and no logs

[simitt]

The Fleet settings UI allows for setting all kind of values in the Elasticsearch output configuration. There is no validation, allowing for any kind of input.

Observed behavior:

• Setting enabled: false: the Elastic Agent kills the sub processes and only logs that it stoped the sub processes. No indicator why they are stopped, no additional entries in the sub process logs.
• Using wrong type, e.g. bulk_max_size: "4s": Elastic Agent keeps restarting the sub processes, which can't start because they all have an invalid configuration. Subprocess logs include detailed information - helpful for debugging

Expected behavior:

• the logs should always contain helpful information - if the Agent stops the sub processes, a reason should be logged.

Ideally there would be some validation preventing to store invalid configurations

• only supported options could be allowed
• loose validation - (most important) supported options are validated, other options are passed on but not validated

I wasn't certain whether this belongs here and/or to Fleet.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[simitt]

Related: when having a managed policy, e.g. on Elastic Cloud, only a subset of the ES output settings should be allowed. For example the host or username and password should not be configurable.