Namespace config values in fleet output

[fearful-symmetry]

CC @cmacknz

Right now, the output config used by elastic-agent is structured like this:

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    api-key: "example-key"
    shipper:
      enabled: true

Under V2, the shipper will receive all the config values under the given config type, like this:

    type: elasticsearch
    hosts: [127.0.0.1:9200]
    api-key: "example-key"
    shipper:
      enabled: true

I'm a little concerned about the lack of namespacing in the output config, as the output config just gets dumped at the root of the config object. The full config used by the elasticsearch output is huge: https://github.com/elastic/elastic-agent-shipper/blob/7bc1701294fd74b99713de8429838c6a55013a0c/output/elasticsearch/config.go#L19 And trying to unpack from the root config object could lead to name collisions. I've already ran into this during testing, as merging the config values set by the -E flag (which elastic-agent uses to set path.data) collides with the path config item in the elasticsearch config. Ideally, the output config received by the shipper would look something like this:

outputs:
  default:
    elasticsearch:
      hosts: ["elasticsearch:9200"]
      api-key: "api_key"
    shipper:
      enabled: true
      queue:
        queue_config: ...
      server:
        server_config: ...

This would make unpacking the config much easier and much less error-prone. I suspect the root of this problem is probably somewhere upstream in fleet, since I'm not totally sure how elastic-agent is generating that outputs section. If this namespace merging is happening upstream before the config gets to elastic-agent, this might be a somewhat hard problem to solve, but I figured I should file the issue anyway.

We might also want to look into having unit.Expected() return return a config object unique to the shipper, or at least specific to the Output type. If the shipper is a known quantity and a core part of the fleet stack, we should make it a first-class citizen of the ecosystem and hard-code portions of the config, and perhaps create a OutputExpectedConfig{} type. The easiest way to do this might be by breaking apart UnitChanges() into input and output-specific streams.

There's also some namespacing issues with unpacking the config used by the shipper's -E flags, when agent sets a -E path.* flag, this will conflict with elasticsearch's path config value.