New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker daemon logs filled node disk space #73208
Comments
/sig cluster-lifecycle |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What happened:
Some pods could not run on the node because there is no space left on
/run
tmpfs."Error":"write /var/run/docker/libcontainerd/737a3f7d9f671903831ed8c5fb4eff4ed4335cc2c81a1c7bff98d890d85cc310/config.json: no space left on device"
What you expected to happen:
Mainly, the pods successfully deployed.
How to reproduce it (as minimally and precisely as possible):
Just have a living node running for a long time to let docker daemon logs fill the disk space.
Anything else we need to know?:
As Docker do not provide any configuration for its daemon logs (Whereas it does for containers logs), Kubernetes should add a new block in logrotate to clean docker daemon logs.
Here are the results for the commands ran inside the Kubernetes node:
Environment:
kubectl version
):Client Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.6-gke.2", GitCommit:"04ad69a117f331df6272a343b5d8f9e2aee5ab0c", GitTreeState:"clean", BuildDate:"2019-01-04T16:21:13Z", GoVersion:"go1.10.3b4", Compiler:"gc", Platform:"linux/amd64"}
Google Kubernetes Engine
uname -a
):Linux gke-***-node-pool-*-********-**** 4.14.65+ #1 SMP Thu Oct 25 10:42:50 PDT 2018 x86_64 Intel(R) Xeon(R) CPU @ 2.20GHz GenuineIntel GNU/Linux
The text was updated successfully, but these errors were encountered: