-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cAdvisor occasionally gets into a state where it has no container metadata #1932
Comments
What version of docker are you using? What docker storage driver are you using? Does it happen consistently, or only occasionally? cAdvisor operates independently of kubelet. The kubelet only queries cAdvisor, and doesn't provide it any information. So it is purely a cAdvisor issue. It seems like the correct behavior here is to continue on, even if we do not have a r/w layer id for the docker container (only drop the "image" field, but not the others). There is also likely an underlying bug with the cAdvisor monitoring for a docker storage driver, or an incompatibility with a more recent docker version |
I think it happens infrequently, although the only evidence is that nobody noticed it happening much. I have set a Prometheus alarm on I don't have any bad nodes at the moment, since we have variously restarted and blown things away. I will check next time it happens. The Docker root dir is nonstandard, but I do have a bunch of files at
|
sorry, typo. Should be |
It's happened again, on one node in each of two clusters, both immediately after a reboot. We do not get the I suspect a race in I'll try setting the |
It happened in our cluster. container / pod name lost in :10255/metrics/cadvisor And lot of kubelet logs:
kubernetes 1.10.2 |
seems a race condition between kubelet start and docker start. --docker-root is deprecated, if docker daemon not running, kubelet failed to read from docker info, will kubelet use default docker graph to init cadvisor? |
@jejer I think it just doesn't work at all if it can't initialize. Where do you see |
BTW after I set To fix the underlying problem we need some alternative to |
@bboreham , cadvisor/container/docker/factory.go Line 62 in f757631
I will try set --docker-root, thank you! |
OK, that is exactly this bug - if Docker is not ready, the info is not available. Note it was known at the time #1275 (comment):
|
added --docker-root for kubelet and never saw this issue again. |
Setting |
--docker-root is already set in my case, but still facing the issue on some nodes. |
Hi @Shivkumar13 did you resolve the issue in the end? We saw this issue if using prometheus |
That issue was not resolved properly at that time, but I rebooted the node as a workaround, and suddenly it started showing metrics. |
k8s 1.13.3 |
I am facing same issue on rhel8 but using standalone cadvisor binary. Cannot use "docker-root" as it is deprecated. Command: cadvisor -disable_root_cgroup_stats=false -docker_only=true -housekeeping_interval=30s -store_container_labels=false Jul 05 23:33:40 cadvisor[3549756]: W0705 23:33:40.810522 3549756 manager.go:1168] Failed to process watch event {EventType:0 Name:/system.slice/46fd7e454d3469b4ab4c96c720cd034774c2c6ef142bc468724f24a7373fa078.service WatchSource:0}: failed to identify the read-write layer ID for container "46fd7e454d3469b4ab4c96c720cd034774c2c6ef142bc468724f24a7373fa078". - open /var/lib/containers/storage/image/overlay/layerdb/mounts/46fd7e454d3469b4ab4c96c720cd034774c2c6ef142bc468724f24a7373fa078/mount-id: no such file or directory |
From kubernetes/kubernetes#39812 (comment)
The high-level symptom is that
curl /metrics
returns blank for image, namespace, etc.example line from broken node:
example line from working node:
We have seen this on two clusters in recent weeks, running k8s v1.9.3.
It appears that kubelet's view of the universe has diverged significantly from Docker's hence it does not have the metadata to tag container metrics. Lots of these in kubelet logs:
Restarting kubelet fixed the issue for the most recent incident. I'm told on another occasion it was necessary to drain, delete all docker files and restart.
The text was updated successfully, but these errors were encountered: