Allow overriding otel exporter and extension configuration for outputs

[swiatekm]

What does this PR do?

Allows otel component configurations generated from output configuration to be overridden. In the output configuration, we have a new optional otel section:

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    api_key: "example-key"
    otel:
      extensions:
        beatsauth:
          timeout: "60s"
      exporter:
        include_source_on_error: true

otel.extensions.<extension_name> contains configuration for the generated extension, while otel.exporter, for the exporter.

These configurations are merged into the generated configuration and take precedence over it. There aren't any validity checks here, so the feature should be used with care. It's only intended as a stopgap measure to mitigate unforeseen problems.

Why is it important?

If it turns out there's an issue with the configuration translation, we want to have an escape hatch to mitigate it for affected customers.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

How to test this PR locally

Build the agent package, then run it with the following configuration:

agent:
  logging:
    to_stderr: true
    level: debug
  monitoring:
    enabled: false
inputs:
- data_stream:
    namespace: default
  id: unique-system-metrics-input
  streams:
  - data_stream:
      dataset: system.cpu
    metricsets:
    - cpu
  type: system/metrics
  use_output: default
  _runtime_experimental: otel
outputs:
  default:
    username: elastic
    password: placeholder
    hosts:
    - 127.0.0.1:9200
    type: elasticsearch
    otel:
      exporter:
        sending_queue:
          queue_size: 50
      extensions:
        beatsauth:
          timeout: "60s"

Then collect diagnostics and look at the final otel configuration. It should contain the output overrides.

Related issues

• Closes [beats receivers] Allow specifying and preserving exporter configuration in Elastic Agent outputs that is preserved through configuration translation #10423

[mergify]

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[leehinman]

Thank You. I really like how straight forward this is.

[cmacknz]

You can Unskip the test below now after enabling telemetry::log_failed_doc_input using this override feature and it should pass.

elastic-agent/testing/integration/ess/beat_receivers_test.go

Lines 1117 to 1133 in a24b6fa

	// TestSensitiveLogsESExporter tests sensitive logs from ex-exporter are not sent to fleet
	func TestSensitiveLogsESExporter(t *testing.T) {
	// The ES exporter logs the original document on indexing failure only if
	// the "telemetry::log_failed_docs_input" setting is enabled and the log level is set to debug.
	info := define.Require(t, define.Requirements{
	Group: integration.Default,
	Local: true,
	Sudo: true,
	OS: []define.OS{
	{Type: define.Windows},
	{Type: define.Linux},
	{Type: define.Darwin},
	},
	Stack: &define.Stack{},
	})
	t.Skip("Enable this test after https://github.com/elastic/elastic-agent/issues/10423 is implemented with appropriate config")

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 42658df

Failed CI Steps

• Win2022:sudo:fleet

History

• 💛 Build #29845 was flaky aa64b35

cc @swiatekm

[cmacknz]

Tested locally, works as expected. Thanks!