systemd 247 journal logs are not processed by splunk/fluentd-hec container #679
Description
What happened:
Latest systemd-journald enabled zstd compression what caused an issue to process binary journals by fluent-plugin-systemd plugin
What you expected to happen:
tag journald.kube:docker
path "/var/log/journal"
matches [{ "_SYSTEMD_UNIT": "docker.service" }]
read_from_head true
<storage>
@type local
persistent true
path /var/log/splunkd-fluentd-journald-docker.pos.json
</storage>
kubelet logs are expected to appear on local storage and Splunk server
How to reproduce it (as minimally and precisely as possible):
- Install K8S nodes based on Flatcar 2905 or any distro with enabled systemd
zstd compression - Do normal deployment of
splunk-connect-for-kuberneteschart with related values to your cluster - Search for logs with tag
journald.kube:dockeron Splunk server - Check /var/log/splunkd-fluentd-journald-docker.pos.json file in the
splunk-connect-for-kubernetes-splunk-kubernetes-loggingcontainer
Anything else we need to know?:
Similar issue reported for
fluent-bit: fluent/fluent-bit#2998
Flatcar: flatcar/Flatcar#328
latest splunk/fluentd-hec container still contains old systemd libraries v 239:
ldd /usr/bin/journalctl
linux-vdso.so.1 (0x00007fff83fd0000)
libsystemd-shared-239.so => /usr/lib/systemd/libsystemd-shared-239.so (0x00007f4d90bf4000)
Environment:
- Kubernetes version (use
kubectl version):
v1.20.10 - Ruby version (use
ruby --version):
Ruby ver insplunk/fluentd-hec:1.2.8containerruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux] - OS (e.g:
cat /etc/os-release):
Flatcar Container Linux by Kinvolk 2905.2.3 (Oklo) - Splunk version:
Version:
8.2.2109 - Splunk Connect for Kubernetes helm chart version:
1.4.10 - Others:
systemd ver247