Skip to content

Non Default collector internal telemetry getting overwritten by the prometheus defaults #4111

Open
@Kidsan

Description

@Kidsan

Component(s)

collector

What happened?

Description

We have the operator version 0.126.0 installed in kubernetes and have set it to deploy our own collector distrubution image, which works. We now want to enable the internal telemetry of the collectors and push to an otlp endpoint, however any configuration we add to the readers field of the collector spec gets overwritten with the default prometheus spec by the operator webhooks. Looking through the code, we found that if telemetry is set, the defaults shouldn't get applied, but this is not the behaviour we are seeing.
Interestingly, we have found if we manually specify the default config, but change the host or port, that change does get applied.

Steps to Reproduce

In an effort to simplify demoing the issue, I have created a test which fails: main...Kidsan:opentelemetry-operator:main

# operator-values.yaml
manager:
  resources:
    limits:
      cpu: 100m
      memory: 256Mi
      ephemeral-storage: 50Mi
    requests:
      cpu: 100m
      memory: 256Mi
      ephemeral-storage: 50Mi

  autoInstrumentationImage:
    java:
      tag: 1.33.6
    nodejs:
      tag: 0.58.0
    python:
      tag: 0.53b1

  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    runAsNonRoot: true
    readOnlyRootFilesystem: true
    seccompProfile:
      type: RuntimeDefault

  podAnnotations:
    "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"

  extraEnvs:
    - name: GOMEMLIMIT
      value: 200MiB

kubeRBACProxy:
  enabled: true

  resources:
    limits:
      cpu: 500m
      memory: 128Mi
    requests:
      cpu: 5m
      memory: 64Mi

  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    runAsNonRoot: true
    readOnlyRootFilesystem: true
    seccompProfile:
      type: RuntimeDefault
    runAsGroup: 1000
    runAsUser: 1000

admissionWebhooks:
  create: true

  certManager:
    enabled: true
    issuerRef: {}

pdb:
  create: false

securityContext:
  fsGroup: 1000
  runAsGroup: 1000
  runAsNonRoot: true
  runAsUser: 1000
# collector.yaml
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  namespace: otel
  name: otel-test
spec:
  mode: deployment
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi
  serviceAccount: collector
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    seccompProfile:
      type: RuntimeDefault
    capabilities:
      drop:
        - ALL
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    processors:
      batch: {}
      batch/test: {}

    exporters:
      otlp:
        endpoint: otelcol:4317

    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp]
      telemetry:
        metrics:
          level: basic
          readers:
            - periodic:
                exporter:
                  otlp:
                    endpoint: otlp_host:4317

Expected Result

We should be able to push telemetry with an otlp exported and the default prometheus telemetry config should not be applied

Actual Result

The default prometheus config is applied and our otlp push config is not.

Kubernetes Version

1.32.4

Operator version

0.126.0

Collector version

custom distribution based on 0.128.0

Environment information

Log output

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:collectorIssues for deploying collectorbugSomething isn't workinggood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions