OTel manager failed: failed to generate otel config: error translating config for output: ... 'hosts' source data must be an array or slice, got string

[cmacknz]

To reproduce in 9.2.1 use the following configuration and run the agent with the command:

ELASTIC_ENDPOINT="myESHost:9200" ./elastic-agent run -e --develop

  outputs:
    default:
      type: elasticsearch
      hosts: 
        - ${ELASTIC_ENDPOINT}
      api_key: "example-key"
      preset: balanced

  agent.monitoring:
    enabled: true
    metrics: true
    logs: true
    use_output: default

  inputs:
    - id: unique-system-metrics-id
      type: system/metrics
      streams:
        - metricset: load
          data_stream.dataset: system.cpu

The agent will become unhealthy with the following error:

message: |-
    OTel manager failed: failed to generate otel config: error translating config for output: monitoring, unit: prometheus/metrics-monitoring, error: failed decoding config. decoding failed due to the following error(s):

    'hosts' source data must be an array or slice, got string

Looking at how Filebeat (or Metricbeat) want the hosts format to be they want it to be a list: https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#hosts-option so at first glance this seems correct:

output.elasticsearch:
  hosts: ["https://myEShost:9200"]

However Beats will happily start if you use a string instead, so we have become more strict than before.
The following config will run where as it's failing here.

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /var/log/*.log

output.elasticsearch:
  hosts: "myESHost:9200"

The workaround is to fix the format of hosts to be a list the way it is expected to be. Since there is only one host the config change below will fix it immediately (hosts: ["${ELASTICSEARCH_HOST}"] also works):

outputs:
    default:
        hosts: 
            - ${ELASTICSEARCH_HOST}

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)