[collector] [receiver/kubeletstats] cannot get "container.id" from receiver/kubeletstats

[fredczj]

Component(s)

receiver/kubeletstats

Describe the issue you're reporting

I have difficulties completing the configuration of otelcol and kubernetes information.

I feel there is something blurry in the metric I collect from my cluster, because, even with a simple configuration, I cannot retrieve container.id from extra_metadata_labels in kubeletstats.

In my understanding, this is the only way to help further k8sattributes to attach all the k8s. resource attributes to the metrics kubeletstats generates (… for pod and container metrics).

What I feel strange is also I cannot connect to the node metric server on port 10250 from node name.

reference

• OVHCloud managed kubernetes - v1.31.1
• otelcol: otel/opentelemetry-collector-k8s:0.120.0

configuration

Kubeletstat receiver

  kubeletstats:
    auth_type: serviceAccount
    collection_interval: 10s
    endpoint: https://${env:K8S_NODE_IP}:10250
    extra_metadata_labels:
    - container.id
    insecure_skip_verify: true
    k8s_api_config:
      auth_type: serviceAccount
    metric_groups:
    - pod
    - node
    - container
    node: ${env:K8S_NODE_NAME}

Full configuration

exporters:
  otlp/tempo:
    endpoint: tempo-distributor:4317
    tls:
      insecure: true
  otlphttp/loki:
    endpoint: http://loki-write:3100/otlp
  otlphttp/mimir:
    endpoint: http://mimir-distributor:8080/otlp


extensions:
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
  k8s_observer:
    auth_type: serviceAccount
    node: ${env:K8S_NODE_NAME}
    observe_nodes: true
    observe_pods: true


processors:
  attributes/addHostName:
    actions:
    - action: insert
      key: host.name
      value: ${env:K8S_NODE_NAME}
  attributes/addk8sHostName:
    actions:
    - action: insert
      key: k8s.node.name
      value: ${env:K8S_NODE_NAME}
  batch: {}
  filter/cyclic:
    logs:
      log_record:
      - attributes["cyclic"] != "true"
    traces:
      span:
      - attributes["cyclic"] != "true"
  filter/standard:
    logs:
      log_record:
      - attributes["cyclic"] == "true"
    traces:
      span:
      - attributes["cyclic"] == "true"
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
  probabilistic_sampler:
    sampling_percentage: 10


receivers:
  kubeletstats:
    auth_type: serviceAccount
    collection_interval: 10s
    endpoint: https://${env:K8S_NODE_IP}:10250
    extra_metadata_labels:
    - container.id
    insecure_skip_verify: true
    k8s_api_config:
      auth_type: serviceAccount
    metric_groups:
    - pod
    - node
    - container
    node: ${env:K8S_NODE_NAME}
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
service:
  extensions:
  - health_check
  - k8s_observer
  pipelines:
    logs:
      exporters:
      - otlphttp/loki
      processors:
      - filter/standard
      - memory_limiter
      - batch
      receivers:
      - otlp
    logs/cyclic:
      exporters:
      - otlphttp/loki
      processors:
      - filter/cyclic
      - probabilistic_sampler
      - memory_limiter
      - batch
      receivers:
      - otlp
    metrics:
      exporters:
      - otlphttp/mimir
      processors:
      - memory_limiter
      - batch
      receivers:
      - otlp
      - kubeletstats
    metrics/kubeletstats:
      exporters:
      - otlphttp/mimir
      processors:
      - attributes/addk8sHostName
      - memory_limiter
      - batch
      receivers:
      - kubeletstats
    traces:
      exporters:
      - otlp/tempo
      processors:
      - filter/standard
      - memory_limiter
      - batch
      receivers:
      - otlp
    traces/cyclic:
      exporters:
      - otlp/tempo
      processors:
      - filter/cyclic
      - probabilistic_sampler
      - memory_limiter
      - batch
      receivers:
      - otlp
  telemetry:
    metrics:
      address: ${env:MY_POD_IP}:8888

Cluster Role

        clusterRole:
          create: true
          name: "opentelemetry-collector-admin"
          clusterRoleBinding:
            name: "opentelemetry-collector-admin"
          rules:
          - apiGroups: [""]
            resources: ["pods", "namespaces"]
            verbs: ["get", "watch", "list"]
          - apiGroups: ["apps"]
            resources: ["replicasets"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["extensions"]
            resources: ["replicasets"]
            verbs: ["get", "list", "watch"]          
          - apiGroups: [""]
            resources: ["nodes/stats"]
            verbs: ["get", "watch", "list"]
          - apiGroups: [""]
            resources: ["events", "namespaces", "namespaces/status", "nodes", "nodes/spec", "pods", "pods/status", "replicationcontrollers", "replicationcontrollers/status", "resourcequotas", "services" ]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["apps"]
            resources: ["daemonsets", "deployments", "replicasets", "statefulsets"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["extensions"]
            resources: ["daemonsets", "deployments", "replicasets"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["batch"]
            resources: ["jobs", "cronjobs"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["autoscaling"]
            resources: ["horizontalpodautoscalers"]
            verbs: ["get", "list", "watch"]
          - apiGroups: [""]
            resources: ["nodes/proxy"]
            verbs: ["get"]

helm additional configuration

I disabled most of presets, using alternateConfig to limit the number of automatic writing…

        extraEnvs:
          - name: K8S_CLUSTER_NAME
            value: dev
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: K8S_NODE_IP
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
          - name: OTEL_RESOURCE_ATTRIBUTES
            value: "host.name=$(K8S_NODE_NAME),host.ip=$(K8S_NODE_IP)"

        presets:
          hostMetrics:
            enabled: false
          kubeletMetrics:
            enabled: true
          kubernetesAttributes:
            enabled: false

Also, checking proxy/stats/summary don't show any container.id. Not sure this is a clue:

metric from /api/v1/nodes/nodeName:10250/proxy/stats/summary

  {
   "podRef": {
    "name": "uidigestv2-listen-dep-647d65d7df-5wrbs",
    "namespace": "uidigestv2",
    "uid": "dcee48f3-d4eb-45a8-9199-46638b60fbc2"
   },
   "startTime": "2025-02-17T18:31:17Z",
   "containers": [
    {
     "name": "uidigestv2-listen",
     "startTime": "2025-02-17T18:31:21Z",
     "cpu": {
      "time": "2025-02-28T16:26:32Z",
      "usageNanoCores": 3909019,
      "usageCoreNanoSeconds": 3074172800000
     },
     "memory": {
      "time": "2025-02-28T16:26:32Z",
      "availableBytes": 124592128,
      "usageBytes": 32694272,
      "workingSetBytes": 32694272,
      "rssBytes": 15892480,
      "pageFaults": 285745,
      "majorPageFaults": 218
     },
     "rootfs": {
      "time": "2025-02-28T16:26:32Z",
      "availableBytes": 87600046080,
      "capacityBytes": 103865303040,
      "usedBytes": 45056,
      "inodesFree": 12591803,
      "inodes": 12902400,
      "inodesUsed": 14
     },
     "logs": {
      "time": "2025-02-28T16:26:36Z",
      "availableBytes": 87600046080,
      "capacityBytes": 103865303040,
      "usedBytes": 28672,
      "inodesFree": 12591803,
      "inodes": 12902400,
      "inodesUsed": 1
     },
     "swap": {
      "time": "2025-02-28T16:26:36Z",
      "swapAvailableBytes": 0,
      "swapUsageBytes": 0
     }
    }
   ],
   "cpu": {
    "time": "2025-02-28T16:26:24Z",
    "usageNanoCores": 3301990,
    "usageCoreNanoSeconds": 3074210878000
   },
   "memory": {
    "time": "2025-02-28T16:26:24Z",
    "usageBytes": 32944128,
    "workingSetBytes": 32944128,
    "rssBytes": 15933440,
    "pageFaults": 289506,
    "majorPageFaults": 218
   },
   "network": {
    "time": "2025-02-28T16:26:29Z",
    "name": "eth0",
    "rxBytes": 2640683558,
    "rxErrors": 0,
    "txBytes": 2327540189,
    "txErrors": 0,
    "interfaces": [
     {
      "name": "eth0",
      "rxBytes": 2640683558,
      "rxErrors": 0,
      "txBytes": 2327540189,
      "txErrors": 0
     }
    ]
   },
   "volume": [
    {
     "time": "2025-02-28T16:25:48Z",
     "availableBytes": 87601025024,
     "capacityBytes": 103865303040,
     "usedBytes": 12288,
     "inodesFree": 12591803,
     "inodes": 12902400,
     "inodesUsed": 5,
     "name": "configfile"
    },
    {
     "time": "2025-02-28T16:25:48Z",
     "availableBytes": 13099241472,
     "capacityBytes": 13099253760,
     "usedBytes": 12288,
     "inodesFree": 1872045,
     "inodes": 1872054,
     "inodesUsed": 9,
     "name": "kube-api-access-fp8hf"
    }
   ],
   "ephemeral-storage": {
    "time": "2025-02-28T16:26:36Z",
    "availableBytes": 87600046080,
    "capacityBytes": 103865303040,
    "usedBytes": 90112,
    "inodesFree": 12591803,
    "inodes": 12902400,
    "inodesUsed": 21
   },
   "process_stats": {
    "process_count": 1
   },
   "swap": {
    "time": "2025-02-28T16:26:24Z",
    "swapUsageBytes": 0
   }
  },

[github-actions]

Pinging code owners:

• receiver/x: @mx-psi @dmathieu

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners:

• receiver/kubeletstats: @dmitryax @TylerHelmuth @ChrsMark

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for receiver/kubeletstats: @dmitryax @TylerHelmuth @ChrsMark. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[fredczj]

🆙

[bacherfl]

Hi @fredczj! I just tried to reproduce this, and it seems in my environment the container ID was retrieved correctly. I also looked at the code, and the container ID should be retrieved from the http://localhost:8080/api/v1/nodes/<nodeName>:10250/proxy/pods endpoint, so that may also be a thing to double check if the container IDs are present there. But since you mentioned that in your case the endpoint could not be reached, I suspect that this may also be the reason why also the container ID is missing. Regarding the connection to the endpoint, do you get any error logs that you can share here?

[fredczj]

Thank you @bacherfl , for your help.

When I run the collector with this configuration, I get no error at all. For instance:

collector log

k8s.pod.cpu.time{host.name=pool2-node-bd9337} 52307.382543
k8s.pod.cpu.utilization{host.name=pool2-node-bd9337} 0.008966156
k8s.pod.filesystem.available{host.name=pool2-node-bd9337} 72038973440
k8s.pod.filesystem.capacity{host.name=pool2-node-bd9337} 103865303040
k8s.pod.filesystem.usage{host.name=pool2-node-bd9337} 23068672
k8s.pod.memory.available{host.name=pool2-node-bd9337} 155467776
k8s.pod.memory.major_page_faults{host.name=pool2-node-bd9337} 404
k8s.pod.memory.page_faults{host.name=pool2-node-bd9337} 145512
k8s.pod.memory.rss{host.name=pool2-node-bd9337} 21696512
k8s.pod.memory.usage{host.name=pool2-node-bd9337} 70025216
k8s.pod.memory.working_set{host.name=pool2-node-bd9337} 22790144
k8s.pod.network.errors{interface=eth0,direction=receive,host.name=pool2-node-bd9337} 0
k8s.pod.network.errors{interface=eth0,direction=transmit,host.name=pool2-node-bd9337} 0
k8s.pod.network.io{interface=eth0,direction=receive,host.name=pool2-node-bd9337} 15211446448
k8s.pod.network.io{interface=eth0,direction=transmit,host.name=pool2-node-bd9337} 26400109207
container.cpu.time{host.name=pool2-node-bd9337} 52307.386236
container.cpu.utilization{host.name=pool2-node-bd9337} 0.008251693
container.filesystem.available{host.name=pool2-node-bd9337} 72038973440
container.filesystem.capacity{host.name=pool2-node-bd9337} 103865303040
container.filesystem.usage{host.name=pool2-node-bd9337} 36864
container.memory.available{host.name=pool2-node-bd9337} 155693056
container.memory.major_page_faults{host.name=pool2-node-bd9337} 404
container.memory.page_faults{host.name=pool2-node-bd9337} 144478
container.memory.rss{host.name=pool2-node-bd9337} 21659648
container.memory.usage{host.name=pool2-node-bd9337} 69799936
container.memory.working_set{host.name=pool2-node-bd9337} 22564864
k8s.pod.cpu.time{host.name=pool2-node-bd9337} 3530.927424
k8s.pod.cpu.utilization{host.name=pool2-node-bd9337} 0.000370928
k8s.pod.filesystem.available{host.name=pool2-node-bd9337} 72038973440
k8s.pod.filesystem.capacity{host.name=pool2-node-bd9337} 103865303040
k8s.pod.filesystem.usage{host.name=pool2-node-bd9337} 53248
k8s.pod.memory.major_page_faults{host.name=pool2-node-bd9337} 510
k8s.pod.memory.page_faults{host.name=pool2-node-bd9337} 98082
k8s.pod.memory.rss{host.name=pool2-node-bd9337} 34795520
k8s.pod.memory.usage{host.name=pool2-node-bd9337} 88502272
k8s.pod.memory.working_set{host.name=pool2-node-bd9337} 68431872
k8s.pod.network.errors{interface=eth0,direction=receive,host.name=pool2-node-bd9337} 0
k8s.pod.network.errors{interface=eth0,direction=transmit,host.name=pool2-node-bd9337} 0
k8s.pod.network.io{interface=eth0,direction=receive,host.name=pool2-node-bd9337} 1406913906
k8s.pod.network.io{interface=eth0,direction=transmit,host.name=pool2-node-bd9337} 1051287380
container.cpu.time{host.name=pool2-node-bd9337} 3530.882229
container.cpu.utilization{host.name=pool2-node-bd9337} 0.0001595
container.filesystem.available{host.name=pool2-node-bd9337} 72038973440
container.filesystem.capacity{host.name=pool2-node-bd9337} 103865303040
container.filesystem.usage{host.name=pool2-node-bd9337} 28672
container.memory.major_page_faults{host.name=pool2-node-bd9337} 510
container.memory.page_faults{host.name=pool2-node-bd9337} 96961
container.memory.rss{host.name=pool2-node-bd9337} 34754560
container.memory.usage{host.name=pool2-node-bd9337} 88276992
container.memory.working_set{host.name=pool2-node-bd9337} 68206592
k8s.pod.cpu.time{host.name=pool2-node-bd9337} 2663.969509
k8s.pod.cpu.utilization{host.name=pool2-node-bd9337} 0.001487267
k8s.pod.filesystem.available{host.name=pool2-node-bd9337} 72038973440
k8s.pod.filesystem.capacity{host.name=pool2-node-bd9337} 103865303040
k8s.pod.filesystem.usage{host.name=pool2-node-bd9337} 86016
k8s.pod.memory.major_page_faults{host.name=pool2-node-bd9337} 172
k8s.pod.memory.page_faults{host.name=pool2-node-bd9337} 478399
k8s.pod.memory.rss{host.name=pool2-node-bd9337} 12988416
k8s.pod.memory.usage{host.name=pool2-node-bd9337} 30277632
k8s.pod.memory.working_set{host.name=pool2-node-bd9337} 29364224
k8s.pod.network.errors{interface=eth0,direction=receive,host.name=pool2-node-bd9337} 0
k8s.pod.network.errors{interface=eth0,direction=transmit,host.name=pool2-node-bd9337} 0
k8s.pod.network.io{interface=eth0,direction=receive,host.name=pool2-node-bd9337} 1064526368
k8s.pod.network.io{interface=eth0,direction=transmit,host.name=pool2-node-bd9337} 1444256244
container.cpu.time{host.name=pool2-node-bd9337} 2663.863857
container.cpu.utilization{host.name=pool2-node-bd9337} 0.001651137
container.filesystem.available{host.name=pool2-node-bd9337} 72038973440
container.filesystem.capacity{host.name=pool2-node-bd9337} 103865303040
container.filesystem.usage{host.name=pool2-node-bd9337} 45056
container.memory.available{host.name=pool2-node-bd9337} 128200704
container.memory.major_page_faults{host.name=pool2-node-bd9337} 172
container.memory.page_faults{host.name=pool2-node-bd9337} 472317
container.memory.rss{host.name=pool2-node-bd9337} 12926976
container.memory.usage{host.name=pool2-node-bd9337} 29999104
container.memory.working_set{host.name=pool2-node-bd9337} 29085696
k8s.pod.cpu.time{host.name=pool2-node-bd9337} 14.434747
k8s.pod.cpu.utilization{host.name=pool2-node-bd9337} 0
k8s.pod.filesystem.available{host.name=pool2-node-bd9337} 72038973440
k8s.pod.filesystem.capacity{host.name=pool2-node-bd9337} 103865303040
k8s.pod.filesystem.usage{host.name=pool2-node-bd9337} 133750784
k8s.pod.memory.major_page_faults{host.name=pool2-node-bd9337} 408
k8s.pod.memory.page_faults{host.name=pool2-node-bd9337} 355556
k8s.pod.memory.rss{host.name=pool2-node-bd9337} 5595136
k8s.pod.memory.usage{host.name=pool2-node-bd9337} 238796800
k8s.pod.memory.working_set{host.name=pool2-node-bd9337} 95649792
k8s.pod.network.errors{interface=eth0,direction=receive,host.name=pool2-node-bd9337} 0
k8s.pod.network.errors{interface=eth0,direction=transmit,host.name=pool2-node-bd9337} 0
k8s.pod.network.io{interface=eth0,direction=receive,host.name=pool2-node-bd9337} 39740500
k8s.pod.network.io{interface=eth0,direction=transmit,host.name=pool2-node-bd9337} 6384617
container.cpu.time{host.name=pool2-node-bd9337} 14.395658
container.cpu.utilization{host.name=pool2-node-bd9337} 0
container.filesystem.available{host.name=pool2-node-bd9337} 72038973440
container.filesystem.capacity{host.name=pool2-node-bd9337} 103865303040
container.filesystem.usage{host.name=pool2-node-bd9337} 133746688
container.memory.major_page_faults{host.name=pool2-node-bd9337} 408
container.memory.page_faults{host.name=pool2-node-bd9337} 354549
container.memory.rss{host.name=pool2-node-bd9337} 5554176
container.memory.usage{host.name=pool2-node-bd9337} 238567424
container.memory.working_set{host.name=pool2-node-bd9337} 95420416
    {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"}

I have checked your observation point, at /proxy/pods, and indeed, I see container ids… 

I feel every points are correct. So I wonder If my problem comes from my understanding of kubeletstats receivers.

When I read pulls node, pod, container, and volume metrics from the API server on a kubelet, I understand this receiver autonomously collects information about pods (from its node), and digest all this.

But today, I run otelcol as an Agent, as a Daemonset, guessing it can get all the information from the pod population on its node from the API. But maybe this one should be run as sidecar, so that it is associated natively with the pods I need to get information from ?

Also, I feel we miss some configuration example on how to convert the container.id into a full labeled metric (pod.name…, host.name…), no?

[bacherfl]

thank you for the additional info, @fredczj - maybe one thing to double check: Can you try to set the verbosity of the debug exporter to detailed? With the normal verbosity, only the metric datapoint attributes will be shown, but not the resource attributes (so also the container.id attribute). I also tried that on my end, and no container.id attribute was present with the normal verbosity, but with detailed, it was logged properly.

Regarding the other question - yes, if the receiver runs on a collector deployed as a daemonset, and the correct node name has been passed to the collector as an env var via the k8s downwards api, the receiver on the collector will get the information of all the pods of the same node. I'm not sure if running as a sidecar will be the right option though, as then I think you might get duplicated metrics, if there are multiple collector instances running as sidecars for pods on the same node.

Regarding the last question, I think those attributes should all be in the resource attributes, which are only logged with the detailed verbosity

[fredczj]

Here is a detailed trace:

----------------------------------------------------------------------------------------------------------------------------------
StartTimestamp: 2025-03-15 07:46:47 +0000 UTC
Timestamp: 2025-04-07 08:41:58.115479463 +0000 UTC
Value: 86180216832
Metric #3
Descriptor:
     -> Name: container.filesystem.capacity
     -> Description: Container filesystem capacity
     -> Unit: By
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host.name: Str(nodepool-0c7a05c6-e922-4609-83-node-fab199)
StartTimestamp: 2025-03-15 07:46:47 +0000 UTC
Timestamp: 2025-04-07 08:41:58.115479463 +0000 UTC
Value: 103865303040
Metric #4
Descriptor:
     -> Name: container.filesystem.usage
     -> Description: Container filesystem usage
     -> Unit: By
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host.name: Str(nodepool-0c7a05c6-e922-4609-83-node-fab199)
StartTimestamp: 2025-03-15 07:46:47 +0000 UTC
Timestamp: 2025-04-07 08:41:58.115479463 +0000 UTC
Value: 45056
Metric #5
Descriptor:
     -> Name: container.memory.available
     -> Description: Container memory available
     -> Unit: By
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host.name: Str(nodepool-0c7a05c6-e922-4609-83-node-fab199)
StartTimestamp: 2025-03-15 07:46:47 +0000 UTC
Timestamp: 2025-04-07 08:41:58.115479463 +0000 UTC
Value: 131055616
Metric #6
Descriptor:
     -> Name: container.memory.major_page_faults
     -> Description: Container memory major_page_faults
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host.name: Str(nodepool-0c7a05c6-e922-4609-83-node-fab199)
StartTimestamp: 2025-03-15 07:46:47 +0000 UTC
Timestamp: 2025-04-07 08:41:58.115479463 +0000 UTC
Value: 149
Metric #7
Descriptor:
     -> Name: container.memory.page_faults
     -> Description: Container memory page_faults
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host.name: Str(nodepool-0c7a05c6-e922-4609-83-node-fab199)
StartTimestamp: 2025-03-15 07:46:47 +0000 UTC
Timestamp: 2025-04-07 08:41:58.115479463 +0000 UTC
Value: 457658
Metric #8
Descriptor:
     -> Name: container.memory.rss
     -> Description: Container memory rss
     -> Unit: By
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host.name: Str(nodepool-0c7a05c6-e922-4609-83-node-fab199)
StartTimestamp: 2025-03-15 07:46:47 +0000 UTC
Timestamp: 2025-04-07 08:41:58.115479463 +0000 UTC
Value: 11505664
Metric #9
Descriptor:
     -> Name: container.memory.usage
     -> Description: Container memory usage
     -> Unit: By
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host.name: Str(nodepool-0c7a05c6-e922-4609-83-node-fab199)
StartTimestamp: 2025-03-15 07:46:47 +0000 UTC
Timestamp: 2025-04-07 08:41:58.115479463 +0000 UTC
Value: 28344320
Metric #10
Descriptor:
     -> Name: container.memory.working_set
     -> Description: Container memory working_set
     -> Unit: By
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host.name: Str(nodepool-0c7a05c6-e922-4609-83-node-fab199)
StartTimestamp: 2025-03-15 07:46:47 +0000 UTC
Timestamp: 2025-04-07 08:41:58.115479463 +0000 UTC
Value: 26230784
    {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"}

----------------------------------------------------------------------------------------------------------------------------------

I don't see container id neither…

Thank you for your help, because I don't really know what to look for…

Below my complete values.yaml:

complete raw value.yaml

*I use alternateConfig instead of config to have a better control on pipelines. *

        mode: daemonset
        image:
          repository: "otel/opentelemetry-collector-k8s"

        resources: 
          limits:
            cpu: 250m
            memory: 512Mi

        command:
          name: "otelcol-k8s"

        securityContext:
          runAsUser: 0
          runAsGroup: 0

        extraEnvs:
          - name: K8S_CLUSTER_NAME
            value: dev
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: K8S_NODE_IP
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
          - name: OTEL_RESOURCE_ATTRIBUTES
            value: "host.name=$(K8S_NODE_NAME),host.ip=$(K8S_NODE_IP)"

        presets:
          hostMetrics:
            enabled: true
          kubeletMetrics:
            enabled: true
          kubernetesAttributes:
            enabled: true
            extractAllPodLabels: true
            extractAllPodAnnotations: true

        podMonitor:
          enabled: true

        ports:
          otlp:
            nodePort: 30317
          otlp-http:
            nodePort: 30318
          metrics:
            enabled: true
          jaeger-compact:
            enabled: false
          jaeger-thrift:
            enabled: false
          jaeger-grpc:
            enabled: false
          zipkin:
            enabled: false

          # - verbs: ["*"]
          #   resources: ["*"]
          #   apiGroups: ["*"]
          # - verbs: ["*"]
          #   nonResourceURLs: ["*"]

        clusterRole:
          create: true
          name: "opentelemetry-collector-admin"
          clusterRoleBinding:
            name: "opentelemetry-collector-admin"
          rules:
          - apiGroups: [""]
            resources: ["pods", "namespaces"]
            verbs: ["get", "watch", "list"]
          - apiGroups: ["apps"]
            resources: ["replicasets"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["extensions"]
            resources: ["replicasets"]
            verbs: ["get", "list", "watch"]          
          - apiGroups: [""]
            resources: ["nodes/stats"]
            verbs: ["get", "watch", "list"]
          - apiGroups: [""]
            resources: ["events", "namespaces", "namespaces/status", "nodes", "nodes/spec", "pods", "pods/status", "replicationcontrollers", "replicationcontrollers/status", "resourcequotas", "services" ]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["apps"]
            resources: ["daemonsets", "deployments", "replicasets", "statefulsets"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["extensions"]
            resources: ["daemonsets", "deployments", "replicasets"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["batch"]
            resources: ["jobs", "cronjobs"]
            verbs: ["get", "list", "watch"]
          - apiGroups: ["autoscaling"]
            resources: ["horizontalpodautoscalers"]
            verbs: ["get", "list", "watch"]
          - apiGroups: [""]
            resources: ["nodes/proxy"]
            verbs: ["get"]

        serviceAccount:
          create: true
          name: "opentelemetry-collector-admin"

        alternateConfig:
          # == KUBERNETES THINGS===================================================================
          extensions:
            health_check:
              endpoint: ${env:MY_POD_IP}:13133
            
          # == RECEIVERS ==========================================================================
          receivers:
            otlp:
              protocols:
                grpc:
                  endpoint: 0.0.0.0:4317
                http:
                  endpoint: 0.0.0.0:4318

            prometheus:
              config:
                scrape_configs:
                - job_name: opentelemetry-collector
                  scrape_interval: 10s
                  static_configs:
                  - targets:
                    - ${env:MY_POD_IP}:8888

            # ++ KUBELET METRICS +++++++++++++++++++++++++++++++++++++++++++++++
            kubeletstats:
              insecure_skip_verify: true
              collection_interval: 30s
              # auth_type: "none"
              auth_type: "serviceAccount"
              endpoint: "${env:K8S_NODE_IP}:10250"
              node: '${env:K8S_NODE_NAME}'
              k8s_api_config:
                auth_type: serviceAccount
              extra_metadata_labels:
                - container.id
              metric_groups: [node, pod, container]

            # # ++ HOST METRICS ++++++++++++++++++++++++++++++++++++++++++++++++++
            # hostmetrics:
            #   collection_interval: 30s
            #   root_path: /hostfs
            #   scrapers:
            #     cpu:
            #       metrics:
            #         system.cpu.logical.count:
            #           enabled: true
            #     disk: null
            #     process:
            #       mute_process_name_error: true
            #       mute_process_exe_error: true
            #       mute_process_io_error: true
            #       mute_process_user_error: true
            #       metrics:
            #         process.cpu.utilization:
            #           enabled: true
            #         process.memory.utilization:
            #           enabled: true
            #         process.disk.io:
            #           enabled: true
            #         process.cpu.time:
            #           enabled: true
            #     filesystem:
            #       metrics:
            #         system.filesystem.utilization:
            #           enabled: true
            #       exclude_fs_types:
            #         fs_types:
            #         - autofs
            #         - binfmt_misc
            #         - bpf
            #         - cgroup2
            #         - configfs
            #         - debugfs
            #         - devpts
            #         - devtmpfs
            #         - fusectl
            #         - hugetlbfs
            #         - iso9660
            #         - mqueue
            #         - nsfs
            #         - overlay
            #         - proc
            #         - procfs
            #         - pstore
            #         - rpc_pipefs
            #         - securityfs
            #         - selinuxfs
            #         - squashfs
            #         - sysfs
            #         - tracefs
            #         match_type: strict
            #       exclude_mount_points:
            #         match_type: regexp
            #         mount_points:
            #         - /dev/*
            #         - /proc/*
            #         - /sys/*
            #         - /run/k3s/containerd/*
            #         - /run/k8s/containerd/*
            #         - /var/lib/docker/*
            #         - /var/lib/kubelet/*
            #         - /snap/*
            #     load: null
            #     memory:
            #       metrics:
            #         system.memory.limit:
            #           enabled: true
            #         system.memory.utilization:
            #           enabled: true
            #     network:
            #       metrics:
            #         system.network.connections:
            #           enabled: true
            #         system.network.io:
            #           enabled: true
            #     paging:
            #       metrics:
            #         system.paging.operations:
            #           enabled: true
            #         system.paging.usage:
            #           enabled: true


          # == PROCESSORS ==========================================================================
          processors:
            # ++ resource saving ++++++++++++++++++++++++++++++++++++++++++++++++
            batch: {}

            memory_limiter:
              check_interval: 5s
              limit_percentage: 80
              spike_limit_percentage: 25

            # ++ adding resource from detected +++++++++++++++++++++++++++++++++++
            resourcedetection:
              detectors: [env, system]
              override: true
              system:
                resource_attributes:
                  host.name:
                    enabled: true
                  host.ip:
                    enabled: true
                  os.type:
                    enabled: false

            k8sattributes:
              extract:
                metadata:
                  - k8s.namespace.name
                  - k8s.deployment.name
                  - k8s.statefulset.name
                  - k8s.daemonset.name
                  - k8s.cronjob.name
                  - k8s.job.name
                  - k8s.node.name
                  - k8s.node.uid
                  - k8s.pod.name
                  - k8s.pod.uid
                  - k8s.pod.start_time
                  - k8s.container.name
              filter:
                node_from_env_var: K8S_NODE_NAME
              passthrough: false
              pod_association:
                - sources:
                    - from: resource_attribute
                      name: k8s.pod.ip
                - sources:
                    - from: resource_attribute
                      name: k8s.pod.uid
                - sources:
                    - from: connection

            # k8sattributes:
            #   auth_type: "serviceAccount"
            #   passthrough: false
            #   filter:
            #     node_from_env_var: K8S_NODE_IP
            #   extract:
            #     metadata:
            #       - k8s.pod.name
            #       - k8s.pod.uid
            #       - k8s.deployment.name
            #       - k8s.namespace.name
            #       - k8s.node.name
            #   pod_association:
            #     - sources:
            #       - from: resource_attribute
            #         name: k8s.pod.ip
            #     - sources:
            #       - from: resource_attribute
            #         name: k8s.pod.uid
            #     - sources:
            #       - from: connection

            # ++ centiloc ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
            probabilistic_sampler:
              sampling_percentage: 10  # only keeps 10% of "cyclic" spans

            # cyclic are all events that are quoted to be "cyclic"
            #   just keep all the events that are cyclic -> filter out all non-cyclic events
            filter/cyclic:
              traces:
                span:
                  - attributes["cyclic"] != "true"
              logs:
                log_record:
                  - attributes["cyclic"] != "true"

            # standard is the opposite of the others above
            filter/standard:
              traces:
                span:
                  - attributes["cyclic"] == "true"
              logs:
                log_record:
                  - attributes["cyclic"] == "true"

            # adding attributes where it needs
            attributes/addHostName:
              actions:
              - key: host.name
                value: "${env:K8S_NODE_NAME}"
                action: insert

            attributes/addk8sHostName:
              actions:
              - key: k8s.node.name
                value: "${env:K8S_NODE_NAME}"
                action: insert


          # == EXPORTERS ===========================================================================
          exporters:
            debug:
              verbosity: detailed
              # sampling_initial: 5
              # sampling_thereafter: 200

            otlphttp/loki:
              endpoint: http://loki-write:3100/otlp
            otlp/tempo:
              endpoint: tempo-distributor:4317
              tls:
                insecure: true
            otlphttp/mimir:
              endpoint: "http://mimir-nginx/otlp"

          # == pipelines ===========================================================================
          service:
            extensions: [health_check]

            telemetry:
              metrics:
                address: ${env:MY_POD_IP}:8888

            pipelines:
              # ++ traces ++++++++++++++++++++++++++++++++++++++++++++++++
              traces/cyclic:
                receivers: [otlp]
                processors: [filter/cyclic, probabilistic_sampler, memory_limiter, batch]
                exporters: [otlp/tempo]

              traces:
                receivers: [otlp]
                processors: [filter/standard, memory_limiter, batch]
                exporters: [otlp/tempo]
            
              # ++ metrics ++++++++++++++++++++++++++++++++++++++++++++++++
              metrics:
                receivers: [otlp]
                processors: [ memory_limiter, batch]
                exporters: [otlphttp/mimir]

              metrics/hostmetrics:
                receivers: [hostmetrics]
                processors: [attributes/addHostName, resourcedetection, memory_limiter, batch]
                exporters: [otlphttp/mimir]

              metrics/kubeletstats:
                receivers: [kubeletstats]
                processors: [attributes/addHostName, k8sattributes, memory_limiter, batch]
                exporters: [otlphttp/mimir, debug]

              # ++ logs ++++++++++++++++++++++++++++++++++++++++++++++++++++
              logs/cyclic:
                receivers: [otlp]
                processors: [filter/cyclic, probabilistic_sampler, memory_limiter, batch]
                exporters: [otlphttp/loki]

              logs:
                receivers: [otlp]
                processors: [filter/standard, memory_limiter, batch]
                exporters: [otlphttp/loki]

[fredczj]

🆙

[ChrsMark]

Hey @fredczj @bacherfl!

From https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/README.md#role-based-access-control:

Additionally, when using extra_metadata_labels or any of the {request|limit}_utilization metrics the processor also needs get permissions for nodes/proxy resources.

So it's 2 things:

1. Ensure for kubeletstats receiver config that extra_metadata_labels is set:

extra_metadata_labels:
  - container.id

2. The Collector has access to the nodes/proxy resources:

- apiGroups: [""]
    resources: ["nodes/proxy"]
    verbs: ["get"]

Here is a Helm configuration I checked on my end:

mode: daemonset

image:
  repository: otel/opentelemetry-collector-contrib
  tag: latest

presets:
  kubeletMetrics:
    enabled: true

clusterRole:
  create: true
  rules:
    - apiGroups: [ "" ]
      resources: [ "nodes/proxy"]
      verbs: [ "get" ]
    - apiGroups:
        - ""
      resources:
        - nodes/stats
      verbs:
        - get
    - nonResourceURLs:
        - "/metrics"
      verbs:
        - get

resources:
  limits:
    cpu: 1
    memory: 1Gi

config:
  exporters:
    debug:
      verbosity: detailed

  receivers:
    kubeletstats:
      auth_type: serviceAccount
      collection_interval: 20s
      endpoint: ${env:K8S_NODE_IP}:10250
      insecure_skip_verify: true
      extra_metadata_labels:
        - container.id

  service:
    extensions: [health_check]
    telemetry:
      logs:
        level: INFO
    pipelines:
      metrics:
        receivers: [ kubeletstats ]
        processors: [ ]
        exporters: [ debug ]

Then I see container.id is properly connected:

...
ResourceMetrics #20
Resource SchemaURL:
Resource attributes:
     -> k8s.pod.uid: Str(fd41dafa-a8ae-4fff-abce-165dda68c05b)
     -> k8s.pod.name: Str(kube-proxy-xqcj5)
     -> k8s.namespace.name: Str(kube-system)
     -> k8s.container.name: Str(kube-proxy)
     -> container.id: Str(5aa104fb64bf2455406b2a4552237edd94901d1135610b3eab779f8e71b3414c)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver 0.123.0
Metric #0
Descriptor:
     -> Name: container.cpu.time
     -> Description: Total cumulative CPU time (sum of all cores) spent by the container/pod/node since its creation
     -> Unit: s
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2025-04-29 08:06:25 +0000 UTC
Timestamp: 2025-04-29 08:20:49.964740792 +0000 UTC
Value: 0.433750
...

If you still have issues please try setting service::telemetry::logs::level to DEBUG in case any debug logs can indicate the issue.

Keep in mind we cannot guarantee or support every single k8s setup unless we have specific steps to reproduce the issue, thank's!