Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs from two different containers of one pod show up in Sumo as coming from one of the containers #2862

Closed
andrzej-stencel opened this issue Feb 10, 2023 · 1 comment · Fixed by #2863
Assignees
Labels
bug Something isn't working

Comments

@andrzej-stencel
Copy link
Contributor

Version: v3.0.1
Kubernetes: v1.25.3
Container runtime: containerd

I have a pod that has two containers in it. I'm seeing the same _sourceName value and container value for logs from both containers, but I would expect them to be different.

Steps to reproduce

  • Create a k8s cluster with e.g. minikube start --kubernetes-version=1.25.3 --container-runtime=containerd

  • Create a pod with two containers

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: logger
    spec:
      selector:
        matchLabels:
          app: logger
      replicas: 1
      template:
        metadata:
          # annotations:
          #   sumologic.com/singleline.sourceCategory: single-line-sc
          labels:
            app: logger
        spec:
          containers:
          - name: singleline
            image: busybox
            args: [/bin/sh, -c, 'i=0; while true; do echo "singleline-$i"; i=$((i+1)); sleep 1; done']
            resources: {}
    
          - name: multiline
            image: busybox
            args: [/bin/sh, -c, 'i=0; while true; do echo "2001-02-03T04:05:06 multiline 1 log $i first line"; echo "  multiline 1 log $i second line"; echo "  multiline 1 log $i third line"; i=$((i+1)); sleep 1; done']
            resources: {}
  • Install collection chart with helm install -n sumo --create-namespace col sumologic/sumologic --version=3.0.1 --values user-values.yaml with the following values file:

    sumologic:
      accessId: xxx
      accessKey: yyy
    
      clusterName: two-containers
    
      events:
        enabled: false
      logs:
        enabled: true
      metrics:
        enabled: false
      traces:
        enabled: false

Expected output

Logs from multiline container should be logged with container=multiline, logs from singleline container should be logged with container=singleline

Actual output

Logs from both containers are logged with container=singleline

image

@andrzej-stencel andrzej-stencel added the bug Something isn't working label Feb 10, 2023
@andrzej-stencel andrzej-stencel self-assigned this Feb 10, 2023
@swiatekm
Copy link

I think this is due to us asking k8s_tagger for the container name here: https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/main/deploy/helm/sumologic/conf/logs/otelcol/config.yaml#L184. If the Pod has multiple containers, this will give the name of the first one, as the processor has no way of telling which one it actually is.

Since we're guaranteed to have the container name here, we should just remove this option.

andrzej-stencel added a commit that referenced this issue Feb 10, 2023
Fixes #2862
The `containerName` metadata extraction gives incorrect results
for pods with more than one containers.
We don't need this extraction anyway, because container logs already have the `k8s.container.name` attribute from log file name.
andrzej-stencel added a commit that referenced this issue Feb 10, 2023
Fixes #2862
The `containerName` metadata extraction gives incorrect results
for pods with more than one containers.
We don't need this extraction anyway, because container logs already have the `k8s.container.name` attribute from log file name.
andrzej-stencel added a commit that referenced this issue Feb 10, 2023
Fixes #2862
The `containerName` metadata extraction gives incorrect results
for pods with more than one containers.
We don't need this extraction anyway, because container logs already have the `k8s.container.name` attribute from log file name.
andrzej-stencel added a commit that referenced this issue Feb 10, 2023
Fixes #2862
The `containerName` metadata extraction gives incorrect results
for pods with more than one containers.
We don't need this extraction anyway, because container logs already have the `k8s.container.name` attribute from log file name.
andrzej-stencel added a commit that referenced this issue Feb 13, 2023
Fixes #2862
The `containerName` metadata extraction gives incorrect results
for pods with more than one containers.
We don't need this extraction anyway, because container logs already have the `k8s.container.name` attribute from log file name.
andrzej-stencel added a commit that referenced this issue Feb 13, 2023
Fixes #2862
The `containerName` metadata extraction gives incorrect results
for pods with more than one containers.
We don't need this extraction anyway, because container logs already have the `k8s.container.name` attribute from log file name.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants