kubectl logs will not output logs after the log file is rotated.

[xyz-li]

What happened?

kubectl logs POD_NAME -f won't output logs after kubelet rotate log filer of the container.

What did you expect to happen?

kubectl output logs correctly after the log file is rotated.

How can we reproduce it (as minimally and precisely as possible)?

1. Add the config containerLogMaxSize: 5Mi to kubelet.yaml. So kubelet will rotate the log soon.
2. Build a image with this program

package main

import (
     "fmt"
     "time"
)

func main() {
     i := 0
     for range time.Tick(10 * time.Microsecond) {
             fmt.Printf("print log to stand out %d\n", i)
             i++
     }
}

3. Create a deploy with the previous image.
4. Run the command kubectl logs POD_NAME -f

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-26T10:58:54Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:51:45Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

None

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
Linux andrewli 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[xyz-li]

/sig node

[xyz-li]

/area kubelet

[SergeyKanzhelev]

/triage accepted
/priority important-longterm

[SergeyKanzhelev]

/assign @xyz-li
since you sent a PR already

[rayjanoka]

Ran into this issue! 😃

[wu0407]

containerd log stuck