invalid capacity 0 on image filesystem warning when starting k3s node #1857
Comments
|
Yes, as the error says, it takes a minute for the stats to be collected. This should be harmless and clear within a few seconds after the node starts. Are you seeing otherwise? |
|
I'd argue that if it's harmless, than it should not be logged as a node warning. If it's expected to take a minute to collect stats, then wait a minute before creating a warning event. Attempting to use node warnings to alert/notify/page our ops staff that there's a potential problem or a concerning event that has occurred that may require attention. The invalid disk capacity warning does not get triggered when rebooting a RKE-based Kubernetes node: |
|
This is unaltered upstream k8s kubelet code: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/images/image_gc_manager.go#L297 . k3s tries not to change upstream code, just build and package it in a format that's easy to run within a single process. You're probably not seeing it on your RKE cluster because it uses docker instead of containerd. I suspect you would see different behavior if you used k3s with docker instead of the default containerd backend. |
|
FYI there's still some long-standing cruft around non-docker (CRI) backends in upstream k8s. See the code linked from kubernetes/kubernetes#51798 (comment) |
|
is there has any answer? help, i meet it also |
|
Ignore it for now. I don't think there's an open issue in k/k for this specific issue yet - feel free to open one if the warning is causing you a lot of heartburn. |
|
Hi, I'm having this issue as well. It's super strange, because I can only add correctly one node to the cluster. The other nodes can be added but without any space available to host pod:
I've tried everything and nothing works. I've even downgraded the kernel version of my raspberry to 4.x and without success |
|
@i5Js that sounds like a different problem - as described above, the warning message is caused by some information not being immediately available at startup, and is cleared several seconds later. It does not prevent anything from working. If your nodes are not working, please open a new issue and fill out the template so that we have all the relevant information from your environment. |
|
I have same problem but with the microk8s. To me was helpful this thread: It seems that problem is hiding somewhere behind count of simltaneous watched files, i slightly increased But root cause is still undetected. |
|
This fixed the Error for me |
|
Our read on this is that upstream k8s is tearing out cadvisor stats from kubelet as part of the docker-shim work. We don't want to pick up fixing this here because it will likely be fixed upstream once that work is completed. |
|
@brandond can you find the issue where they're working cadvisor stats collection and link it here? |
|
Hello to everyone I was getting the same error when I installed kubernetes cluster via kubeadm. After reading all the comments on the subject, I thought that the problem might be caused by containerd and the following two commands solved my problem, maybe it can help
|
|
mine got the same error and has a problem. |
unfortunately, these command is not working on my K3S |
|
Any update on this issue? I'm also getting There's no feedback on the suggestion by @ruffst , and my k3s environment does not have Other debug info: |
|
No. This will need to be fixed upstream. I suspect it will be fixed when we upgrade to containerd v1.6 with the cri-api v1 changes but I haven't confirmed that yet. The |
|
Thanks @brandond . I should have asked my question in a more helpful way. Is there anything that I (or we, the community) can do as a workaround / patch until the upstream addresses this item? I really do not understand the root cause, nor it's effects, so I'm stumped. If I could understand a bit more, I may be able to help. If it is certain limitations in using k3s on Raspbian, Raspberry Pi, or My guess, at this stage, is that my k3s deployment is either limited and/or fragile. I'm unsure if I'm in a safe state to continue deploying resources to this installation. |
|
Are you talking about the |
|
Sorry, yes, I was referring to I came to this page initially, because I saw the error |
|
They are both safe to ignore. |
I had a 'NotReady' status for master/worker nodes. This worked perfectly @selcukmeral. Thanks a lot! |
this worked for me also, after kubeadm init, then apply calico then restart containerd and kubelet is only way the node became Ready |
|
This message will continue as long as the kubelet considers this a reportable error. The reason has been laid out in the comments above. |
|
I am going to close this. The message is irritating but not critical, and there does not seem to be any fix for it short of waiting a full stats-collection interval after starting containerd, before starting the kubelet. As that would extend K3s startup by ~20 seconds, we are not going to do that. |
Version:
K3s arguments:
/usr/local/bin/k3s server --cluster-cidr 172.16.0.0/16 --service-cidr 192.168.0.0/16 --cluster-dns 192.168.0.10 --no-deploy traefik --kube-apiserver-arg feature-gates=ServiceTopology=true,EndpointSlice=true
Describe the bug
When starting a node, getting the following warning in the k8s events:
To Reproduce
systemctl restart k3sExpected behavior
Should not see this warning.
Actual behavior
FYI all node events:
Additional context / logs
Seeing these messages in the logs:
gz#10525
The text was updated successfully, but these errors were encountered: