How to see container, pod, namespace tabs in google cloud logs exporter?

[owensulei3841]

Hi All,

I created a opentelemetry collector sidecar container in my GKE. I try to use this collector to scrap logs from my application then exporter it to google logging dashboard. Right now I'm able to see the logs showing in "Logs Exporter", but I can't see tabs like "container name" , "pod name" on the left panel. Do I need to provide some resource attributes to make this work?

This is my logs explorer page:
You can see no "pod name", "container name" below the "Error" Tab.

This is my collector yaml file:

 receivers:
    otlp:
      protocols:
        grpc:
    filelog:
      include: [/opt/yahoo/o11y/log/*.log]
      start_at: end 
      operators:
        - type: regex_parser
          regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) (?P<sev>[A-Z]*) (?P<msg>.*)$$'
          timestamp:
            parse_from: attributes.time
            layout: '%Y-%m-%d %H:%M:%S.%f'
          severity:
            parse_from: attributes.sev
 
 processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 70
      spike_limit_percentage: 30
    resourcedetection/gcp:
      detectors: [ env, gcp ]
      timeout: 2s
      override: true
    batch:
      send_batch_size: 800
      timeout: 5s
    k8sattributes:
      auth_type: "serviceAccount"
      passthrough: true
      filter:
        node_from_env_var: K8S_NODE_NAME
      extract:
        metadata:
          - k8s.deployment.name
          - k8s.namespace.name
          - k8s.node.name
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.pod.start_time
          - k8s.container.name
    transform:
      error_mode: ignore
      log_statements:
        - context: resource
          statements:
            - set(attributes["exporter_namespace_name"], attributes["service.name"])
            - set(attributes["exporter_node_name"], attributes["k8s.node.name"])
            - delete_key(attributes, "k8s.node.name")
            - set(attributes["exporter_pod_name"], attributes["k8s.pod.name"])
            - delete_key(attributes, "k8s.pod.name")
            - set(attributes["exporter_container_name"], attributes["k8s.container.name"])
            - delete_key(attributes, "k8s.container.name")

  exporters:
    file:
      path: ./o11y-collector.log
    logging:
      verbosity: basic
    googlecloud:
      log:
        default_log_name: collector-exported-log
        resource_filters: [ prefix: "cloud", prefix: "host"]

  service:
    telemetry:
      logs:
        level: "info"
        output_paths: [./telemetry.log]
    pipelines:
      logs/file:
        receivers: [ filelog ]
        processors: [ memory_limiter, k8sattributes, resourcedetection/gcp, transform, batch ]
        exporters: [ googlecloud ]

Dear googlecloud exporter contributors, please help me! Thanks a lot!!

cc @aabmass @dashpole @jsuereth @punya @damemi

[dashpole]

Can you try with the logging exporter, and check the resource attributes the log records have?

To map to k8s_container, you need to set the following resource attributes:

• cloud.platform == gcp_kubernetes_engine
• cloud.region or cloud.availability_zone
• k8s.cluster.name
• k8s.namespace.name
• k8s.pod.name
• k8s.container.name

The gcp resource detector should be detecting k8s.cluster.name and cloud.region/cloud.availability_zone. I suspect your k8sattributes processor isn't doing what you want (although you can confirm). I would recommend using the k8s downward API, similar to https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/detectors/gcp#setting-kubernetes-attributes. Then, the env resourcedetector will add those. Additionally, your sidecars don't need to watch k8s resources from the kubernetes API server, which would cause issues at scale.