GoogleManagedPrometheus not having the Resource Attributes as Resource/Metric labels in Cloud Monitoring Console

[avijitsarkar123]

Component(s)

exporter/googlemanagedprometheus

Describe the issue you're reporting

I have the receiver k8s_cluster collecting k8s cluster metrics, I am exporting them to googlemanagedprometheus, I have also added the logging exporter that shows the resource attributes for the metrics but in the GCP Cloud Monitoring console the resource labels are only having: project_id,location,cluster, namespace, job, instance

Metric #4
Descriptor:
     -> Name: k8s.container.restarts
     -> Description: How many times the container has restarted in the recent past. This value is pulled directly from the K8s API and the value can go indefinitely high and be reset to 0 at any time depending on how your kubelet is configured to prune dead containers. It is best to not depend too much on the exact value but rather look at it as either == 0, in which case you can conclude there were no restarts in the recent past, or > 0, in which case you can conclude there were restarts in the recent past, and not try and analyze the value beyond that.
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-07-21 05:50:55.499885716 +0000 UTC
Timestamp: 2023-07-21 05:55:55.489937456 +0000 UTC
Value: 0
ResourceMetrics #11
Resource SchemaURL: 
Resource attributes:
     -> k8s.pod.uid: Str(5d0e0477-64d3-4a9e-a0f6-aa1cfa24df21)
     -> k8s.pod.name: Str(kube-dns-576f648469-8kz5r)
     -> k8s.node.name: Str(gke-dev-gcp-us-west1-dev-gcp-uswe1-np-81098db8-m4c3)
     -> k8s.namespace.name: Str(kube-system)
     -> opencensus.resourcetype: Str(container)
     -> container.id: Str(716f263b569fb9bc2f617066a342f3198e42e90be0a945ee29100168800de856)
     -> k8s.container.name: Str(dnsmasq)
     -> container.image.name: Str(sha256)
     -> container.image.tag: Str(7d5f5b64708dde4099625f23e85936a902c7d9ed393d43076818518e8f7e2f9b)
     -> cloud.provider: Str(gcp)
     -> cloud.account.id: Str(xxx-dev-09qb)
     -> cloud.platform: Str(gcp_kubernetes_engine)
     -> cloud.region: Str(us-west1)
     -> k8s.cluster.name: Str(dev-gcp-us-west1)
     -> host.id: Str(4774348630754743591)
     -> k8s.deployment.name: Str(kube-dns)
     -> k8s.pod.start_time: Str(2023-07-13 12:28:53 +0000 UTC)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/k8sclusterreceiver 0.81.0

I tried adding them using transform but didn't work, this is what my otel pipeline looks like:

receivers:
      k8s_cluster:
        collection_interval: 30s
      # The hostmetrics receiver is required to get correct infrastructure metrics in Datadog.

    processors:
      groupbyattrs:
        keys:
          - namespace
          - cluster
          - location
          - pod_name
      
      k8sattributes:
        auth_type: "serviceAccount"
        passthrough: false
        filter:
          node_from_env_var: KUBE_NODE_NAME
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.namespace.name
            - k8s.node.name
            - k8s.pod.start_time
            - k8s.container.name
        pod_association:
          # below association takes a look at the datapoint's k8s.pod.ip resource attribute and tries to match it with
          # the pod having the same attribute.
          - sources:
            - from: resource_attribute
              name: k8s.pod.ip
          # below association matches for pair `k8s.pod.name` and `k8s.namespace.name`
          - sources:
            - from: resource_attribute
              name: k8s.pod.name
            - from: resource_attribute
              name: k8s.namespace.name

      resourcedetection:
        # detect cluster name and location
        detectors: [gcp]
        timeout: 5s
        override: false
            
      batch:
        # batch metrics before sending to reduce API usage
        send_batch_max_size: 100
        send_batch_size: 100
        timeout: 5s

      memory_limiter:
        # drop metrics if memory usage gets too high
        check_interval: 1s
        limit_percentage: 85
        spike_limit_percentage: 20
      
      transform:
        metric_statements:
          - context: datapoint
            statements:
              - set(attributes["pod_name"], attributes["k8s.pod.name"])
              - set(attributes["container_name"], attributes["k8s.container.name"])

    exporters:
      logging:
        verbosity: detailed
      googlemanagedprometheus:
        project: {{ .Values.provider.id }}

    service:
      telemetry:
        logs:
          level: debug
      pipelines:
        metrics:
          receivers: [k8s_cluster]
          processors: [batch, memory_limiter, resourcedetection, groupbyattrs, k8sattributes, transform]
          exporters: [logging, googlemanagedprometheus]  

In the below, if I replace attributes["k8s.pod.name"] with some text like TEST then it shows up in the GCP Cloud Monitoring as label but when trying to read from resource attributes its not working.

transform:
        metric_statements:
          - context: datapoint
            statements:
              - set(attributes["pod_name"], attributes["k8s.pod.name"])
              - set(attributes["container_name"], attributes["k8s.container.name"])

My requirement is just to have all the resource attributes as metrics/resource labels in GCP Cloud Monitoring for those metrics.

[github-actions]

Pinging code owners:

• exporter/googlemanagedprometheus: @aabmass @dashpole @jsuereth @punya @damemi @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[avijitsarkar123]

So in the below code attributes["k8s.pod.name"] should be resource.attributes["k8s.pod.name"] , after fixing this the issue got resolved.

But is there a better way to automatically add the resource attributes to metrics labels instead of manually doing it? I came across a property resource_to_telemetry_conversion that does the same for AmazonManagedPrometheus

transform:
        metric_statements:
          - context: datapoint
            statements:
              - set(attributes["pod_name"], attributes["k8s.pod.name"])
              - set(attributes["container_name"], attributes["k8s.container.name"])

[dashpole]

After #24392, you will be able to use resource_filters to configure that behavior.

[damemi]

duplicate of #21654

[crobert-1]

@avijitsarkar123 Can you confirm your issue has been resolved? If so, you can close this issue.

[crobert-1]

/label -needs-triage

[crobert-1]

/label waiting-for-author

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/googlemanagedprometheus: @aabmass @dashpole @jsuereth @punya @damemi @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Closing. You can use metric.resource_filters on the GMP exporter.