'kubectl logs -f insert_pod_here' stops reporting when log file rotates

[rouilj]

Environmental Info:
K3s Version:

k3s version v1.27.7+k3s2 (575bce76)
go version go1.20.10

Node(s) CPU architecture, OS, and Version:

Linux odroid6 4.9.337-92 #1 SMP PREEMPT Thu Jul 20 05:16:44 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:

Server/agent running on a single node.

Describe the bug:

kubectl logs -f pod stops reporting output when the log file rotates.

Further details can be seen at: stern/stern#287. That issue diagnosed it as a manifestation of: kubernetes/kubernetes#115701 which is solved in kubernetes/kubernetes#115702.

Steps To Reproduce:

• Installed K3s: Using standard method from web page using wget ....| sh.

stern/stern#287 has a kind config for reproducing the issue. Set up a pod that outputs data to the log, set the max log size to a small number. Run kubectl logs -f <podname> wait for the log file to rotate and see the log following stop.

I am using the following in /etc/rancher/k3s/config.yml:

kubelet-arg:
  - container-log-max-files=4
  - container-log-max-size=500Ki

set the max size lower to trigger the issue faster.

Expected behavior:

I expect the logs to continue to be scrolled to the screen as long as the pod exists.

Actual behavior:

After some period of time corresponding to the rotation time of the logs output stops.

Additional context / logs:

[brandond]

the apiserver streams logs from the kubelet, which in turn streams them from the container runtime implementation. None of these are something we customize in k3s, so the issue will need to be resolved elsewhere.

As the issue you linked appears to be fixed in Kubernetes 1.29, and not backported to prior minors, I am going to close this as resolved upstream. Our v1.29.0+k3s1 release is currently underway, you can check out #9052 to see how this is progressing.

[brandond]

I will also note that building log collection frameworks around things that stream from kubctl logs -f isn't a great idea. If you are building a log collection framework, a better idea is to read directly from /var/log/pods on the nodes.

[rouilj]

I'm not doing log collection per-se, I was troubleshooting load balanced pods. While doing that I spent a lot of time spinning when one of the pods appeared to stop receiving connections. What had happened was the log had rotated and the connections were happening but I wasn't seeing them 8-(.