Stern stops tailing logs after a while (possible log file rotation??) #287
Comments
|
Just a followup. It does look related to log file rotation. My My guess is kubectl was displaying the logs for the other pod which was rotated 23 minutes earlier. When stern stopped displaying logs, I looked at the logs in the log directory. Stern stopped displaying the log lines from the log it was reporting about 4 lines before the end of the rotated log file. The log file was rotated from This looks like stern doesn't have the equivalent of |
|
Hi, Thank you for reporting the issue. I have confirmed that both kubectl and stern stop tailing logs when log file rotations occur in Kubernetes v1.28.0. I believe the issue is caused by kubernetes/kubernetes#115701, which has been addressed in kubernetes/kubernetes#115702. I have verified that the issue has been resolved in Kubernetes v1.29.0.
Steps to reproduce
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.28.0@sha256:b7a4cad12c197af3ba43202d3efe03246b3f0793f162afb40a33c923952d5b31
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
"container-log-max-size": "1Ki"
apiVersion: v1
kind: Pod
metadata:
name: p
spec:
terminationGracePeriodSeconds: 0
containers:
- name: p
image: busybox
command: ["/bin/sh", "-c", "while true; do date; sleep 0.01; done"]
When we create a Kubernetes v1.29.0 cluster with the same config, the issue is no longer reproduced. (A node image for Kubernetes v1.29.0 is not yet released, so we need to build a v1.29.0 image using |
|
Thanks for the followup and the reproducer. I have reported this upstream using your diagnosis info to k3s-io/k3s#9060. Hopefully, they can use the same or similar fix. I assume once this is fixed stern will work as expected. |
|
Did this problem manifest as if the container had exited cleanly? For instance: I interpreted this as the pod having finished successfully, except that there was no reason for it to have terminated there. When checking logs in GCP, I found that it hadn't actually terminated, and continued to emit additional logs after this point. Restarting stern with the same query showed the above logs, and what came after them, without the above indication that the pod had terminated. I'm currently on 1.25, and I'm not sure if the linked kubernetes bug is older than 1.28. I'll open a new issue if this doesn't sound like the same behavior. Thanks for your time. |
|
@irons Hi, As far as I confirmed, the logs just stopped without the The |
|
I have updated k3s since I opened this and I do see the - trailing line: restarting stern picked up after the last timestamp logged for both pods. |
|
Thanks, and also yikes. I don't want to have to attempt restarting after each pod exit to see if it was kidding, and we won't get to 1.29 any time soon. I'll try again with high verbosity and see if I can get more information. |
What happened:
I start stern and it start tailing the log file for my pods. I am using:
After some period of time (a few hours), the logs stop being reported.
What you expected to happen:
Logs should continue until I stop stern or the pods are stopped.
How to reproduce it (as minimally and precisely as possible):
I'm not sure exactly. I run two pods that emit output with the following config.yaml:
The deployment etc can be obtained from: https://wiki.roundup-tracker.org/DeployWithK3s#preview
I am trying to determine if
kubectl logs -f deployment/roundup-demois stopping at the same point.Anything else we need to know?:
When stern is properly tailing the pod logs, I have tested that stern follows the new and old log files
when deployment rollout is restarted.
Environment:
stern --version): 1.27.0cat /etc/os-release): Ubuntu 20.04.6 LTSThe text was updated successfully, but these errors were encountered: