Ensure monitoring the Otel collector never runs in a beat process

[swiatekm]

What does this PR do?

It ensures that the prometheus metrics input we use to monitor the otel collector always runs in said otel collector. The reason this has to be the case is that the input relies on an environment variable we only inject into the otel collector process, and not into the metricbeat process.

The fix is quite hacky, but I think this is acceptable for two reasons:

• We're redoing how this self-monitoring works in 9.3, and the problem will disappear.
• This is a symptom of a broader issue, where the self-monitoring config generation doesn't know which runtime a component will actually run in - just what we want it to run in. The fix for this is more involved and probably shouldn't go into a patch release.

Why is it important?

If a user has a monitoring output which isn't supported by the otel runtime, their agent will become unhealthy for no good reason.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

Build agent locally and run it using the following configuration, which falls back to the process runtime, as the indices setting on the elasticsearch output isn't supported for otel:

agent:
  logging:
    to_stderr: true
  monitoring:
    enabled: true
inputs:
- data_stream:
    namespace: default
  id: unique-system-metrics-input
  streams:
  - data_stream:
      dataset: system.cpu
    metricsets:
    - cpu
  type: system/metrics
  use_output: default
outputs:
  default:
    username: elastic
    password: xxx
    hosts:
    - 127.0.0.1:9200
    type: elasticsearch
    indices: []

You should see a log line about the prometheus/metrics-monitoring output being skipped, and it shouldn't show up in status.

Related issues

• Closes [beats receivers] host parsing failed for prometheus-collector: error parsing URL: parse "http://localhost:EDOT_COLLECTOR_METRICS_PORT": invalid port ":EDOT_COLLECTOR_METRICS_PORT" after host #11169

[mergify]

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pchila]

Couple of questions, nothing major

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: f9294cb

Failed CI Steps

• Win2025:sudo:default
• arm:sudo: upgrade

History

• 💔 Build #30451 failed d52cfcd
• 💔 Build #30428 failed e65d38b
• 💔 Build #30389 failed 7070c29

cc @swiatekm