Create Kubernetes logs input-plugin

[megastef]

Logagent works fine with its docker-plugin on Kubernetes, when docker is used as container runtime. When containerd is used (e.g. IBM Cloud) the log collection via /var/run/docker.sock does not work.

I suggest creating a Kuberntes specific log collection plugin to avoid any trouble when alternative container runtimes are used in Kubernetes.

[otisg]

Would it be possible to have a generic container plugin that can detect the runtime and automatically use/do the right thing under the hood without the person having to configure Logagent explicitly for Kubernetes or non-Kubernetes env?

[megastef]

The start script can deal with different environments to configure LA.
We can detect the runtime environment and disable docker-collector when docker socket is not available, or running in Kuberntes.

Well, the k8s user must mount /var/log into LA container to make the k8s plugin work.

Therefore we will need Kubernetes specific installation instructions, and use k8s API instead of docker API.

[megastef]

The following DaemonSet should work on IBM Cloud.

• support for containerd log headers
• file collection and enrichment with podName, namespace, containerName, containerId

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: st-logagent
spec:
  selector:
    matchLabels:
      app: st-logagent
  template:
    metadata:
      labels:
        app: st-logagent
    spec:
      #nodeSelector: {}
      hostNetwork: true
      dnsPolicy: "ClusterFirst"
      restartPolicy: "Always"
      containers:
      - name: st-logagent
        image: sematext/logagent:latest
        # image: sematext/logagent:2.0.143
        imagePullPolicy: "Always"
        env:
          - name: LOG_GLOB
            value: "/var/log/containers/*.log;/var/log/*.log"
          - name: LOGS_TOKEN
            value: "YOUR_SEMATEXT_LOGS_TOKEN"
          - name: LOGS_RECEIVER_URL
            value: "https://logsene-receiver.sematext.com"
        volumeMounts:
          - mountPath: /var/log
            name: var-log
          - mountPath: /var/data/kubeletlogs
            name: kubeletlogs
          - mountPath: /etc/localtime
            name: localtime
        securityContext:
          privileged: true
      volumes:
        - name: var-log
          hostPath:
            path: /var/log
        - name: kubeletlogs
          hostPath:
            path: /var/data/kubeletlogs
        - name: localtime
          hostPath:
            path: /etc/localtime

Note, we are still testing and have seen some crashes, which might be unrelated to the new Kubernetes input plugin.

[megastef]

It turns out that Logagent restarts are caused by tail-forever on Node 10 mingqi/tailF#12 - sematext/logagent:2.0.146 ignores the uncaughtException until the issue gets fixed by a new release of tail-forever. We will try to create a pull-request or publish a new version of tail-forever node module.

The final daemonset for a more stable deployment requires v2.0.146 or later:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: st-logagent
spec:
  selector:
    matchLabels:
      app: st-logagent
  template:
    metadata:
      labels:
        app: st-logagent
    spec:
      #nodeSelector: {}
      hostNetwork: true
      dnsPolicy: "ClusterFirst"
      restartPolicy: "Always"
      containers:
      - name: st-logagent
        image: sematext/logagent:2.0.146
        imagePullPolicy: "Always"
        env:
          - name: LOG_GLOB
            value: "/var/log/containers/*.log;/var/log/*.log"
          - name: LOGS_TOKEN
            value: "YOUR_SEMATEXT_LOGS_TOKEN"
          - name: LOGS_RECEIVER_URL
            value: "https://logsene-receiver.sematext.com"
        volumeMounts:
          - mountPath: /var/log
            name: var-log
          - mountPath: /var/data/kubeletlogs
            name: kubeletlogs
          - mountPath: /etc/localtime
            name: localtime
        securityContext:
          privileged: true
      volumes:
        - name: var-log
          hostPath:
            path: /var/log
        - name: kubeletlogs
          hostPath:
            path: /var/data/kubeletlogs
        - name: localtime
          hostPath:
            path: /etc/localtime```

[megastef]

Version sematext/loagagent:2.0.149 uses Node 8 and runs stable without exceptions.

[megastef]

Problem with Node 10 was fixed mingqi/tailF#12

[megastef]

Documentation: https://sematext.com/docs/logagent/installation-docker/#kubernetes-with-containerd-and-ibm-cloud