[8.19] (backport #10992) Allow overriding otel exporter and extension configuration for outputs

[mergify]

What does this PR do?

Allows otel component configurations generated from output configuration to be overridden. In the output configuration, we have a new optional otel section:

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    api_key: "example-key"
    otel:
      extensions:
        beatsauth:
          timeout: "60s"
      exporter:
        include_source_on_error: true

otel.extensions.<extension_name> contains configuration for the generated extension, while otel.exporter, for the exporter.

These configurations are merged into the generated configuration and take precedence over it. There aren't any validity checks here, so the feature should be used with care. It's only intended as a stopgap measure to mitigate unforeseen problems.

Why is it important?

If it turns out there's an issue with the configuration translation, we want to have an escape hatch to mitigate it for affected customers.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

How to test this PR locally

Build the agent package, then run it with the following configuration:

agent:
  logging:
    to_stderr: true
    level: debug
  monitoring:
    enabled: false
inputs:
- data_stream:
    namespace: default
  id: unique-system-metrics-input
  streams:
  - data_stream:
      dataset: system.cpu
    metricsets:
    - cpu
  type: system/metrics
  use_output: default
  _runtime_experimental: otel
outputs:
  default:
    username: elastic
    password: placeholder
    hosts:
    - 127.0.0.1:9200
    type: elasticsearch
    otel:
      exporter:
        sending_queue:
          queue_size: 50
      extensions:
        beatsauth:
          timeout: "60s"

Then collect diagnostics and look at the final otel configuration. It should contain the output overrides.

Related issues

• Closes [beats receivers] Allow specifying and preserving exporter configuration in Elastic Agent outputs that is preserved through configuration translation #10423

---

This is an automatic backport of pull request #10992 done by [Mergify](https://mergify.com).

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)