The old pod log file is not deleted from the /var/log/pods/ directory

[Black-max12138]

What happened?

In the /var/log/pods/ directory, many old log files are not deleted in a timely manner. As a result, the disk space is used up.

What did you expect to happen?

I think there should be only 1 old log file in this path.

How can we reproduce it (as minimally and precisely as possible)?

I don't know how this happened.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# 1.28

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[pranav-pandey0804]

/sig node

[pranav-pandey0804]

/sig storage

[pranav-pandey0804]

/remove-sig node

[pranav-pandey0804]

hi @Black-max12138 , thanks for reporting the issue,
could you verify if the containerLogMaxSize and containerLogMaxFiles settings are correctly configured or not?

[pranav-pandey0804]

If these settings are not configured correctly then kubelet may not delete the old log files efficiently because these setting dictate the maximum size each log file can reach before being rotated and the max log files that can be retained !

[pranav-pandey0804]

you can also refer to this page for more details !

[Black-max12138]

hi @Black-max12138 , thanks for reporting the issue, could you verify if the containerLogMaxSize and containerLogMaxFiles settings are correctly configured or not?

containerLogMaxSize and containerLogMaxFiles are using default values

[Black-max12138]

If these settings are not configured correctly then kubelet may not delete the old log files efficiently because these setting dictate the maximum size each log file can reach before being rotated and the max log files that can be retained !

So what is the possibility of this happening if the parameters are set correctly?

[tamilselvan1102]

Generally once the POD is terminated & a new instance is started, the log files associated with the old pod get deleted.

[jsafrane]

AFAIK container logs are managed by sig-node, not by sig-storage
/sig node
/remove-sig node

[k8s-ci-robot]

@jsafrane: Those labels are not set on the issue: sig/node

In response to this:

AFAIK container logs are managed by sig-node, not by sig-storage
/sig node
/remove-sig node

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tamilselvan1102]

hi @Black-max12138 , thanks for reporting the issue, could you verify if the containerLogMaxSize and containerLogMaxFiles settings are correctly configured or not?

By default, Kubernetes truncates the pod’s container log if it reaches 10 MB.
if you want to change the file size, you can set through containerLogMaxSize parameter in the kubelet configuration.
These parameters are not related to delete/persist the log file.

[haircommander]

/assign @harche