feat: support custom otel collector config (BETA)

[wrn14897]

plus the fix to reduce bloat in opamp agent logs

Users should be able to mount the custom otel collector config file and add/overrider receivers, processors and exporters
For example:

receivers:
  hostmetrics:
    collection_interval: 5s
    scrapers:
      cpu:
      load:
      memory:
      disk:
      filesystem:
      network:
# override the default processors
processors:
  batch:
    send_batch_size: 10000
    timeout: 10s
  memory_limiter:
    limit_mib: 2000
service:
  pipelines:
    metrics/hostmetrics:
      receivers: [hostmetrics]
      # attach existing processors
      processors: [memory_limiter, batch]
      # attach existing exporters
      exporters: [clickhouse]

This will add a new hostmetrics receiver + metrics/hostmetrics pipeline and update existing batch + memory_limiter processors

WARNING: This feature is still in beta, and future updates may change how it works, potentially affecting compatibility

Ref: HDX-1865

[changeset-bot]

🦋 Changeset detected

Latest commit: 8a04890

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@hyperdx/api	Patch
@hyperdx/app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
hyperdx-v2-oss-app	Ready	Preview	Comment	Aug 18, 2025 9:22pm

[wrn14897]

running the log rotator script along with the main otelcol process to rotate agent.log file since its size grows rapidly

[github-actions]

Stably Runner - Test Suite - 'Smoke Test'

Test Suite Run Result: 🟢 Success (4/4 tests passed) [dashboard]

---

This comment was generated from stably-runner-action

[knudtty]

I see a lot of cases like this. Seems like we are telling the Otel collector to look at it's own env vars to find these values. But with a deployment like this, the opampController.ts could just fill these out and provide the Otel Collector config with fully formed config. Which is preferable? It's almost like a templating problem, should the opampController provide a template or a fully rendered config?

[wrn14897]

In the OSS setup, it makes more sense to provide the CH credentials on the otel collector side, which also has a smaller surface area from a security perspective. We also want to be cautious about changes that affect the Helm chart. This PR is purely a refactor and does not introduce any breaking changes.

[knudtty]

Tested locally, all seems well