[collector] [receiver/kubeletstats] cannot get "container.id" from receiver/kubeletstats

[fredczj]

Component(s)

receiver/kubeletstats

Describe the issue you're reporting

I have difficulties completing the configuration of otelcol and kubernetes information.

I feel there is something blurry in the metric I collect from my cluster, because, even with a simple configuration, I cannot retrieve container.id from extra_metadata_labels in kubeletstats.

In my understanding, this is the only way to help further k8sattributes to attach all the k8s. resource attributes to the metrics kubeletstats generates (… for pod and container metrics).

What I feel strange is also I cannot connect to the node metric server on port 10250 from node name.

reference

• OVHCloud managed kubernetes - v1.31.1
• otelcol: otel/opentelemetry-collector-k8s:0.120.0

configuration

Kubeletstat receiver

  kubeletstats:
    auth_type: serviceAccount
    collection_interval: 10s
    endpoint: https://${env:K8S_NODE_IP}:10250
    extra_metadata_labels:
    - container.id
    insecure_skip_verify: true
    k8s_api_config:
      auth_type: serviceAccount
    metric_groups:
    - pod
    - node
    - container
    node: ${env:K8S_NODE_NAME}

Full configuration

exporters:
  otlp/tempo:
    endpoint: tempo-distributor:4317
    tls:
      insecure: true
  otlphttp/loki:
    endpoint: http://loki-write:3100/otlp
  otlphttp/mimir:
    endpoint: http://mimir-distributor:8080/otlp


extensions:
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
  k8s_observer:
    auth_type: serviceAccount
    node: ${env:K8S_NODE_NAME}
    observe_nodes: true
    observe_pods: true


processors:
  attributes/addHostName:
    actions:
    - action: insert
      key: host.name
      value: ${env:K8S_NODE_NAME}
  attributes/addk8sHostName:
    actions:
    - action: insert
      key: k8s.node.name
      value: ${env:K8S_NODE_NAME}
  batch: {}
  filter/cyclic:
    logs:
      log_record:
      - attributes["cyclic"] != "true"
    traces:
      span:
      - attributes["cyclic"] != "true"
  filter/standard:
    logs:
      log_record:
      - attributes["cyclic"] == "true"
    traces:
      span:
      - attributes["cyclic"] == "true"
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
  probabilistic_sampler:
    sampling_percentage: 10


receivers:
  kubeletstats:
    auth_type: serviceAccount
    collection_interval: 10s
    endpoint: https://${env:K8S_NODE_IP}:10250
    extra_metadata_labels:
    - container.id
    insecure_skip_verify: true
    k8s_api_config:
      auth_type: serviceAccount
    metric_groups:
    - pod
    - node
    - container
    node: ${env:K8S_NODE_NAME}
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
service:
  extensions:
  - health_check
  - k8s_observer
  pipelines:
    logs:
      exporters:
      - otlphttp/loki
      processors:
      - filter/standard
      - memory_limiter
      - batch
      receivers:
      - otlp
    logs/cyclic:
      exporters:
      - otlphttp/loki
      processors:
      - filter/cyclic
      - probabilistic_sampler
      - memory_limiter
      - batch
      receivers:
      - otlp
    metrics:
      exporters:
      - otlphttp/mimir
      processors:
      - memory_limiter
      - batch
      receivers:
      - otlp
      - kubeletstats
    metrics/kubeletstats:
      exporters:
      - otlphttp/mimir
      processors:
      - attributes/addk8sHostName
      - memory_limiter
      - batch
      receivers:
      - kubeletstats
    traces:
      exporters:
      - otlp/tempo
      processors:
      - filter/standard
      - memory_limiter
      - batch
      receivers:
      - otlp
    traces/cyclic:
      exporters:
      - otlp/tempo
      processors:
      - filter/cyclic
      - probabilistic_sampler
      - memory_limiter
      - batch
      receivers:
      - otlp
  telemetry:
    metrics:
      address: ${env:MY_POD_IP}:8888

Cluster Role

        clusterRole:
          create: true
          name: "opentelemetry-collector-admin"
          clusterRoleBinding:
            name: "opentelemetry-collector-admin"
          rules:
          - apiGroups: [""]
            resources: ["pods", "namespaces"]
            verbs: ["get", "watch", "list"]
          - apiGroups: ["apps"]
            resources: ["replicasets"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["extensions"]
            resources: ["replicasets"]
            verbs: ["get", "list", "watch"]          
          - apiGroups: [""]
            resources: ["nodes/stats"]
            verbs: ["get", "watch", "list"]
          - apiGroups: [""]
            resources: ["events", "namespaces", "namespaces/status", "nodes", "nodes/spec", "pods", "pods/status", "replicationcontrollers", "replicationcontrollers/status", "resourcequotas", "services" ]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["apps"]
            resources: ["daemonsets", "deployments", "replicasets", "statefulsets"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["extensions"]
            resources: ["daemonsets", "deployments", "replicasets"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["batch"]
            resources: ["jobs", "cronjobs"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["autoscaling"]
            resources: ["horizontalpodautoscalers"]
            verbs: ["get", "list", "watch"]
          - apiGroups: [""]
            resources: ["nodes/proxy"]
            verbs: ["get"]

helm additional configuration

I disabled most of presets, using alternateConfig to limit the number of automatic writing…

        extraEnvs:
          - name: K8S_CLUSTER_NAME
            value: dev
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: K8S_NODE_IP
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
          - name: OTEL_RESOURCE_ATTRIBUTES
            value: "host.name=$(K8S_NODE_NAME),host.ip=$(K8S_NODE_IP)"

        presets:
          hostMetrics:
            enabled: false
          kubeletMetrics:
            enabled: true
          kubernetesAttributes:
            enabled: false

Also, checking proxy/stats/summary don't show any container.id. Not sure this is a clue:

metric from /api/v1/nodes/nodeName:10250/proxy/stats/summary

  {
   "podRef": {
    "name": "uidigestv2-listen-dep-647d65d7df-5wrbs",
    "namespace": "uidigestv2",
    "uid": "dcee48f3-d4eb-45a8-9199-46638b60fbc2"
   },
   "startTime": "2025-02-17T18:31:17Z",
   "containers": [
    {
     "name": "uidigestv2-listen",
     "startTime": "2025-02-17T18:31:21Z",
     "cpu": {
      "time": "2025-02-28T16:26:32Z",
      "usageNanoCores": 3909019,
      "usageCoreNanoSeconds": 3074172800000
     },
     "memory": {
      "time": "2025-02-28T16:26:32Z",
      "availableBytes": 124592128,
      "usageBytes": 32694272,
      "workingSetBytes": 32694272,
      "rssBytes": 15892480,
      "pageFaults": 285745,
      "majorPageFaults": 218
     },
     "rootfs": {
      "time": "2025-02-28T16:26:32Z",
      "availableBytes": 87600046080,
      "capacityBytes": 103865303040,
      "usedBytes": 45056,
      "inodesFree": 12591803,
      "inodes": 12902400,
      "inodesUsed": 14
     },
     "logs": {
      "time": "2025-02-28T16:26:36Z",
      "availableBytes": 87600046080,
      "capacityBytes": 103865303040,
      "usedBytes": 28672,
      "inodesFree": 12591803,
      "inodes": 12902400,
      "inodesUsed": 1
     },
     "swap": {
      "time": "2025-02-28T16:26:36Z",
      "swapAvailableBytes": 0,
      "swapUsageBytes": 0
     }
    }
   ],
   "cpu": {
    "time": "2025-02-28T16:26:24Z",
    "usageNanoCores": 3301990,
    "usageCoreNanoSeconds": 3074210878000
   },
   "memory": {
    "time": "2025-02-28T16:26:24Z",
    "usageBytes": 32944128,
    "workingSetBytes": 32944128,
    "rssBytes": 15933440,
    "pageFaults": 289506,
    "majorPageFaults": 218
   },
   "network": {
    "time": "2025-02-28T16:26:29Z",
    "name": "eth0",
    "rxBytes": 2640683558,
    "rxErrors": 0,
    "txBytes": 2327540189,
    "txErrors": 0,
    "interfaces": [
     {
      "name": "eth0",
      "rxBytes": 2640683558,
      "rxErrors": 0,
      "txBytes": 2327540189,
      "txErrors": 0
     }
    ]
   },
   "volume": [
    {
     "time": "2025-02-28T16:25:48Z",
     "availableBytes": 87601025024,
     "capacityBytes": 103865303040,
     "usedBytes": 12288,
     "inodesFree": 12591803,
     "inodes": 12902400,
     "inodesUsed": 5,
     "name": "configfile"
    },
    {
     "time": "2025-02-28T16:25:48Z",
     "availableBytes": 13099241472,
     "capacityBytes": 13099253760,
     "usedBytes": 12288,
     "inodesFree": 1872045,
     "inodes": 1872054,
     "inodesUsed": 9,
     "name": "kube-api-access-fp8hf"
    }
   ],
   "ephemeral-storage": {
    "time": "2025-02-28T16:26:36Z",
    "availableBytes": 87600046080,
    "capacityBytes": 103865303040,
    "usedBytes": 90112,
    "inodesFree": 12591803,
    "inodes": 12902400,
    "inodesUsed": 21
   },
   "process_stats": {
    "process_count": 1
   },
   "swap": {
    "time": "2025-02-28T16:26:24Z",
    "swapUsageBytes": 0
   }
  },

[github-actions]

Pinging code owners:

• receiver/x: @mx-psi @dmathieu

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners:

• receiver/kubeletstats: @dmitryax @TylerHelmuth @ChrsMark

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for receiver/kubeletstats: @dmitryax @TylerHelmuth @ChrsMark. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[fredczj]

🆙