[receiver/kubeletstats] validate command expects local certificate files.

[tredman]

Component(s)

receiver/kubeletstats

What happened?

Description

Hello! My team has a CI pipeline step that runs otelcol-contrib validate against collector configurations that we maintain. When upgrading from otelcol-contrib 0.97.0 to 0.98.0, validation started failing with the following error:

2024-04-25T12:49:57.208-0700 info service@v0.98.0/telemetry.go:55 Setting up own telemetry...
2024-04-25T12:49:57.208-0700 info service@v0.98.0/telemetry.go:97 Serving metrics {"address": ":8888", "level": "Basic"}
Error: failed to build pipelines: failed to create "kubeletstats" receiver for data type "metrics": cert path /var/run/secrets/kubernetes.io/serviceaccount/ca.crt could not be read: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory
2024/04/25 12:49:57 collector server run finished with error: failed to build pipelines: failed to create "kubeletstats" receiver for data type "metrics": cert path /var/run/secrets/kubernetes.io/serviceaccount/ca.crt could not be read: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory

It appears to be looking for certificate files that would normally only exist if we were running the command in a kubernetes environment. While we run the collector itself in kubernetes, the validation step is run in our CI environment as well as locally on our desktops. Unless I am misunderstanding the purpose of validate, I think it's reasonable to assume that this subcommand should run and be able to validate the configuraiton without dependencies like this.

Steps to Reproduce

1. Create an otel collector configuration with a kubeletstats receiver configuration. I've included a sample of our kubeletstats config here:

    receivers:
      # Pulls node, pod, container, and volume metrics from the API server on each kubelet
      # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/README.md
      kubeletstats:
        auth_type: serviceAccount
        collection_interval: 30s
        endpoint: https://${K8S_NODE_NAME}:10250
        metric_groups:
          - node
          - pod
          - container
        metrics:
          k8s.pod.uptime:
            enabled: true

2. With otelcol-contrib v0.97.0 or earlier, run otelcol-contrib validate --config <path_to_config>. The configuration will validate.

3. With otelcol-contrib v0.98.0, re-run otelcol-contrib validate --config <path_to_config> and observe the error reported above.

Expected Result

validate subcommand should run without external certificate dependencies.

Actual Result

validate fails with this error:

2024/04/25 12:49:57 collector server run finished with error: failed to build pipelines: failed to create "kubeletstats" receiver for data type "metrics": cert path /var/run/secrets/kubernetes.io/serviceaccount/ca.crt could not be read: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory

Collector version

0.98.0

Environment information

Environment

We've reproduced this problem in macOS 14.4.1 as well as in our CI environment running in alpine linux.

OpenTelemetry Collector configuration

receivers:
  # Pulls node, pod, container, and volume metrics from the API server on each kubelet
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/README.md
  kubeletstats:
    auth_type: serviceAccount
    collection_interval: 30s
    endpoint: https://${K8S_NODE_NAME}:10250
    metric_groups:
      - node
      - pod
      - container
    metrics:
      k8s.pod.uptime:
        enabled: true

processors:
  batch:
    timeout: 200ms # default
    send_batch_size: 8192 # default
    send_batch_max_size: 0 # default

extensions:
  # Needed to use IAM to authenticate to the prom API.
  sigv4auth/amp:
    service: "aps"
    region: "us-east-1"

exporters:
  prometheusremotewrite/amp:
    endpoint: "https://fakeURL"
    auth:
      authenticator: sigv4auth/amp

service:
  extensions:
    - sigv4auth/amp
  pipelines:
    metrics:
      receivers:
        - kubeletstats
      processors:
        - batch
      exporters:
        - prometheusremotewrite/amp

Log output

2024-04-25T12:49:57.208-0700	info	service@v0.98.0/telemetry.go:55	Setting up own telemetry...
2024-04-25T12:49:57.208-0700	info	service@v0.98.0/telemetry.go:97	Serving metrics	{"address": ":8888", "level": "Basic"}
Error: failed to build pipelines: failed to create "kubeletstats" receiver for data type "metrics": cert path /var/run/secrets/kubernetes.io/serviceaccount/ca.crt could not be read: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory
2024/04/25 12:49:57 collector server run finished with error: failed to build pipelines: failed to create "kubeletstats" receiver for data type "metrics": cert path /var/run/secrets/kubernetes.io/serviceaccount/ca.crt could not be read: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory

Additional context

When testing my example config you may also run into this other issue I filed here: #32214

[github-actions]

Pinging code owners:

• receiver/kubeletstats: @dmitryax @TylerHelmuth

%s See Adding Labels via Comments if you do not have permissions to add labels yourself.

[frzifus]

This may require a somewhat deeper investigation. At last I wasnt able to find any obvious change. Between 0.97.0 an 0.98.0.

• https://github.com/open-telemetry/opentelemetry-collector-contrib/commits/v0.98.0/receiver/kubeletstatsreceiver
• https://github.com/open-telemetry/opentelemetry-collector-contrib/commits/v0.98.0/internal/kubelet
• https://github.com/open-telemetry/opentelemetry-collector/commits/v0.98.0/receiver/scraperhelper (The inherited config validate method)

cc @dmitryax @TylerHelmuth

[TylerHelmuth]

I think it is very likely this issue is related to open-telemetry/opentelemetry-collector#10031 and/or open-telemetry/opentelemetry-collector#9257

[tredman]

Confirming this is resolved for me in 0.100.0 - thank you!