Does cri-dockerd support journald logging ??

[jeffgtxjava]

Wanted to leverage cri-dockerd as CRI for k8s.

We tested the shim code and noticed kubectl logs doesnt work anymore when docker was configured with journald as logging driver.

Is there an option I need to enable to make cri-dockerd as proxy and leverage the functionality of docker as is??

Error logs:

-sh-4.2# 
kubectl logs test-9ng99 -n test -c test -f
failed to try resolving symlinks in path "/var/log/pods/test_test-9ng99_e5c59673-3af9-4b01-a764-4ebb6adc4313/test/5.log": lstat /var/log/pods/test_test-9ng99_e5c59673-3af9-4b01-a764-4ebb6adc4313/test/5.log: no such file or directory
-sh-4.2# 

docker info

-sh-4.2# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 139
  Running: 59
  Paused: 0
  Stopped: 80
 Images: 734
 Server Version: 20.10.9
 Storage Driver: btrfs
  Build Version: Btrfs v4.9.1
  Library Version: 102
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
 runc version: v1.1.1-0-g52de29d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.62.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 48
 Total Memory: 125.6GiB
 Name: test.local.net
 ID: 6IQV:5TXZ:ATQ5:FQK7:HSDQ:OULR:BAAN:CRME:K2GO:GXAB:U4OL:IRT7
 Docker Root Dir: /cowdata/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels: 
 Experimental: false
 Insecure Registries:
  test.loca.net:5000
  127.0.0.0/8
 Live Restore Enabled: false

[jeffgtxjava]

used the default cri-docker.service file shipped in the RPM

[evol262]

If docker is configured with journald logging, it completely bypasses the normal CRI mechanisms.

The best practice way of managing this is to use something like grafana-agent or vector or your log aggregator of choice, and have multiple sinks. Either forwarding to Graylog, Loki, Logstash, or a process which reads from the system and forwards to journald.

In particular, logging to journald localizes logs for any given container/pod to the system they're running on, which is orthogonal to k8s logging (accessible from anywhere). The only reasonable mechanism to have this work would be to have kube-api able to poll from journald on every single worker node, which is not practical for a lot of reasons.

I'm happy to help you find an alternative logging structure which can give something like the best of both worlds.

[SaberStrat]

Hm, kubernetes’ original Dockershim worked with journald logging though—at least as of 1.23.x.

Did they deprecate more than just the CRI-fication of Docker in 1.24?

[evol262]

cri-docker runs as a systemd unit, which logs to journald/journalctl (from the daemon itself). The answer to log collection is exactly the same as the comment from Jun 7.

[SaberStrat]

Sorry, I didn’t mean cri-dockerd’s logs, I mean the Pod/Container logs.

I meant that kubelet <=1.23 with a Docker Runtime with the Logger Driver set to journald was able to somehow provide container logs upon kubectl logs ….

I absolutely understand from your earlier comment that containers logging to journald isn’t conforming to CRI and that there’s several ways to setup logging the CRI way. Docker itself isn’t CRI conformant either, and so shouldn't be used as the underlying Container Runtime, and yet here we are and you guys made an adapter (which I'm really glad exists :)).

So I was just wondering what part about the now-abandoned Dockershim or Docker-compatibility in the kubernetes/kubelet Code was responsible for kubelet logs's magic/un-CRI compatibility to a Docker Runtime using journald as the Logging Engine.
An abandoned part that's beyond cri-dockerd's scope or purpose (because I thought this project wants to provide a full third-party replacement for the Dockershim)? Or am I completely misunderstanding how kubectl logs used to work with Docker writing container logs to journal?

[evol262]

The now-abandoned Dockershim is this codebase. Upstream k8s was nice enough to extract it and hand it over, and it saw some refactoring to make it more sense as a standalone codebase, but nothing else was removed. I haven't walked back whatever else happened in kubectl/etc, but this is dockershim in 99% of ways.

AFAIK (and judging from the message), kubectl logs isn't doing anything particularly intelligent there, and I wouldn't expect it to, honestly. It's trying to read a symlink (which, after rotation, would probably be /var/lib/docker/containers/xxxxxxx/....) which doesn't point anywhere after rotation (there is likely to be [012345...].log). I would guess, without explicitly testing, that configuring dual logging in docker/daemon.json to log to both json-file (the default) and journald would give the desired behavior.