Error: failed to start containers: kubelet #13314
Comments
|
I got this when adding a new node. I started the agent container as per the command in the UI and it seems to have started a second agent container named |
|
What is the output of |
|
Here are some logs from my terminal: There was no container named |
|
Ok and |
|
I'm afraid it is too late to do that since that container is gone. Here is some more logs from what I did next when I removed the stuck container: |
|
I got same error when adding second node to k8s cluster. here's some logs. |
|
I've never seen anything interesting in the initial rancher agent's logs. |
|
Interestingly I also got this when adding my second node, but the third was added fine. Basically what I did was follow the quickstart to set everything up on a single VM, then add a second VM as worker node and got the error. Then added a third node as worker and it went well. |
|
Hmm... I will try setup machine again(now using baremetal). |
|
I'm not sure if bare metal or VM matters. I think I got similar errors when trying to deploy it on bare metal (my laptop) |
|
Hi, I also had this error because my VMs had the same name. After changing the name and cleaning entire VM (volume prune, system prune etc.) everything went back to normal. |
|
I also had the same issue. But it turned out my |
|
in my case, problem solved by changing name. thanks! |
|
So it is currently not possible to run the |
|
I am getting the same error. I just tried to deploy a cluster. All the masters that I started have the 'share-mnt' containers saying: All I ran was the docker run script generated by rancher to start the nodes. |
|
I have seen this bug many times in many different places/circumstances. The only way I've found to fix it is by destroying the cluster and recreating it. |
|
This cluster I just created in OpenStack right before running the docker command on each host. If the error is for something else like network or hostname like said above, it would be nice if it said so. Right now with the cannot find kublet error isn't very helpful since it was able to pull hyperkube and start other containers. |
|
I get the same error, then I destroy the cluster and recreating it from the latest version. Then I add 3 nodes, and they are ok. the system version: CentOS Linux release 7.4.1708 (Core) the docker version: Server Version: 17.03.2-ce the rancher images; REPOSITORY TAG IMAGE ID CREATED SIZE
|
|
I'm getting same error, I tried destroying and recreating the cluster but no luck. I'm using 2.0.2 version of rancher.
|
|
I used registry which had authentication enabled, after making the rancher repo public, my issue is resolved. |
|
Same here on Ubuntu 18.04 with docker 17.03.2-ce (tried with 18.X but no luck). |
|
Hi, same problem here DinD (Alpine Linux) Docker 18.03.1-ce Rancher 2.0.2 |
|
I have run into the same issue as well. I dont understand where it is finding the container id from, i have checked the local mounts on the node. I was testing some scenarios so tried to re-register a node, after cleaning the local mounts. I have also tried:
|
|
the actual error comes from /usr/bin/share-root.sh I am not really sure if this is an issue with the logic, but no container named |
|
The root cause can either be found in the other |
|
I am seeing the same issue when attempting to re-register a node. The steps I use are:
In this case the rancher server reports: 2018/06/27 14:45:35 [ERROR] ClusterController c-v2998 [cluster-agent-controller] failed with : could not contact server: Get https://x.x.x.x:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused The node has only 2 containers: share-mnt: Error response from daemon: {"message":"No such container: kubelet"} The only way to proceed at this point is stop all activity on the node, delete the node from the rancher ui, and recreate the cluster and re register. Running server 2.0.4 with only the one node, Docker version 17.03.2-ce on Ubuntu 16.04. |
|
+1 for seeing the same thing on a brand new Rancher 2.0 deployment with a new set of CentOS7 nodes |
|
Same for me. |
|
I got this error on my first HA RKE install when adding my first cluster. Wiping nodes and re-creating the cluster had no effect. Removing the cluster and restarting rancher server nodes fixed it for me |
|
I solved removing the hostname from 127.0.0.1 on /etc/hosts and including the FQDN also. So the machine now is responding the hostname and FQDN to the eth0 not loopback. Resuming: 192.168.1.X hostname FQDN |
|
I hit this issue as well. In my case it started when the last node of my Here's what I did:
Note that this will delete all the Docker volumes, take care if you have
Note that official docs on node cleanup recommend also removal of
Inside the container: Now in the editor I found the key
I also tried to just clean-up the node and register it again, but it always Given the above, I think Rancher doesn't handle well the case when all the |
|
I just created a new local VM with Ubuntu 18.04 to test if rancher 2.2.6 is 'stable' now. But I can't add the first worker with all 3 roles on my single machine setup because of this issue. |
|
|
|
Man.. there sure are a lot of posts here. |
|
Rancher stable 2.2.8 also found this issue()
docker ps -a
|
|
I have run into the same issue as well , but solved.
rancher server log: Found that the images address is more http. |
|
Spent days tracking this backwards, looking for answers. Anyone still getting this error, you should first note that the I have noticed that if you fail to provision a node, it will always fail well before it pulls the kubernetes container images, due to the cluster recognizing the node but assigning it a different certificate (or something), leaving it hung. I solved it by wiping my Rancher server and starting over. This has absolutely got to be fixed properly, somehow. HOWEVER... If you actually do get a container named from what I can tell, it means you're running a system where the OS is on a tmpfs volume (either diskless, or not installed to disk yet). There are other errors similar to this in the Docker and Kubernetes forums. If I get to the bottom of where it happens, I'll open a ticket. |
|
Got to the bottom of it. I was running diskless iPXE booted RancherOS. Although rancher-agent started up, by the time it started installing Kubernetes onto the machine, it exhausted something in the tmpfs file system, causing a failure so bad that nothing shows up when you "ls -al /proc", and "df" or "du" would just fail. Three different perfectly good servers with almost exactly the same results. There may be a kernel setting that would expand the OS file system somehow, but by installing to hard drive and booting there instead, all the issues went away. Still worth noting that if you are watching the logs of share-mnt while spinning up a node, quite a number of errors will pop up while k8s is installing, even on a completely healthy node. |
|
I bumped into the very same issue today. I have basically been trying to add a worker node to rancher all morning. It seems in my tests I have been impatiently adding/removing nodes from the cluster by just deleting them from the UI. My suspicion is that etcd cluster got dirty as a result. I did not rebuild the cluster from scratch, as in my case I knew exactly when the tests started. I merely restored an earlier etcd snapshot, tried again and it all worked as expected. |
Were you able to get worker nodes stood up on an iPXE booted RancherOS? I am currently trying to boot 10 nodes this way (without local storage), and it does not seem to be working (seeing the same errors you were). Seeing the following in docker.log: |
|
@Just-Insane Never got it working. From what I could tell, the tmpfs is exhausted of inodes or something, to the extent the OS stopped working. Someone with more time on their hands can maybe sort it out for the rest of us, but I just threw a small HDD in each box and installed ROS there instead. Worked fine. Ultimately gave up on iPXE because of it, unfortunately, and ROS as well, because 99% of the value of it, to me at least, was to treat nodes as extremely dumb cattle. |
|
Ah, that's unfortunate. My only thought is maybe switching to K3sOS, but that is rather poorly documented, especially for iPXE booting. Yea, the only reason I'm using iPXE right now is because of these node's lack of local storage, otherwise, I'd just install an OS and it'd be fine. Further to your note on dumb cattle, I am not sure rancher would handle non-persistent nodes trying to join with existing node names anyways, so that might not even work. |
|
I had set up my DHCP to assign the same IP address to the machine's MAC addresses, then ran an internal web server that trivially returned a hostname according to the requestor's IP address. The nodes would boot, curl the server for its name, then set it right up front in the cloud init. It's an afternoon to set up, but not hard. Overall didn't seem worth pursuing it, given the failure to run entirely in RAM. I am running a k3os cluster in AWS, and probably would not recommend it for running iPXE, since it really wants an external high availability MySQL setup that it can keep its brains in. RDS is good for that, but running locally.... ehh. |
|
Maybe I am misunderstanding, but k3os requires an external HA MySQL setup? I thought it was just a lighter weight version of RancherOS. Hopefully someone on the OS team is able to take a look at that issue and provide a workaround or resolution. What did you end up doing OS wise with your hosts you were trying to get working diskless? |
|
I'm not sure if k3os and k3s is the same thing (I suspect not, on reflection), but to configure an HA Rancher management cluster, they recommend using k3s rather than ROS, which does need something to store shared data into. The easy way is MySQL, the hard way is etcd. https://rancher.com/docs/rancher/v2.x/en/installation/k8s-install/kubernetes-rke/ I ended up dropping an older server SAS drives about 80gb, which is plenty large for an OS. I dropped Ubuntu 18.04 on it and did a simple install. Life became much easier after that. |
|
Oh, my Rancher master cluster is on separate nodes, and my master nodes for this cluster have local storage, so I am fine on that front. Ah, I might have to end up adding local storage I guess. Not looking forward to that cost lol. |
|
I'm having the same issue: I added a third node to a cluster, worked great. I've checked the diskspace, all ok. Also the hostname is correct and set up correctly in the hosts file. Docker ps output when this happens: I also tried changing the hostname but no joy there. Any pointers would be greatly appreciated. And I join the chorus of those who ask for a better error message that lets us actually determine what's wrong. EDIT: I think I'll create a separate issue because this thread is filled with lots of unrelated issues, and it's not even sure it's the same as the original issue. |
|
Have the same issue after every reboot worker node. Every time help docker prune, rm folders and readd rancher-client. Wait solution how fix it |
|
Just FYI I haven't gotten to investigate or pull logs yet, but I will do that when I get some time. I've opened another issue for this which has been closed because I can't touch that system at the moment. |
|
kubelet failing also is an indicator that your node was added to the rancher cluster before, failed for some other reason, then you tried to add it again. If you don't use the cleanup script, you probably haven't gotten rid of a few items that make rancher misinterpret the node as re-joining a cluster rather than being a node new to the cluster. Hunt for it, you'll find it. |
|
I got the same error while creating a new cluster and add a node. And then resolve it! Here are some info: Check steps:
Maybe you need to clean up the new cluster and node after failed options.
Finally, up the new cluster and GLHF! Please let me know if anything can be optimized, thanks! |
|
This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions. |
and others
Rancher versions:
rancher/server or rancher/rancher: rancher/rancher:latest@sha256:38839bb19bdcac084a413a4edce7efb97ab99b6d896bda2f433dfacfd27f8770
rancher/agent or rancher/rancher-agent:
rancher/rancher-agent:v2.0.0
Infrastructure Stack versions:
whatever the defaults are
Docker version: (
docker version,docker infopreferred)Client:
Version: 17.03.2-ce
API version: 1.27
Go version: go1.8.3
Git commit: f5ec1e2-snap-345b814
Built: Thu Jun 29 23:40:29 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.2-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.8.3
Git commit: f5ec1e2-snap-345b814
Built: Thu Jun 29 23:40:29 2017
OS/Arch: linux/amd64
Experimental: false
Operating system and kernel: (
cat /etc/os-release,uname -rpreferred)4.15.0-20-generic
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
ssdnodes/custom cluster
Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)
single node
Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
can't set up node
Steps to Reproduce:
snap install docker --channel=17.03/stable
mkdir /etc/kubernetes
sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.0.0 --server https://rancher.sixcorners.info --token abc --ca-checksum xyz --worker
docker logs -f share-mnt
Results:
Found state.json: 931882e24ff0ef67b0e8744dbf1f7e04fd68afe714a29a2522293312824f3c51
time="2018-05-06T06:09:15Z" level=info msg="Execing [/usr/bin/nsenter --mount=/proc/21787/ns/mnt -F -- /var/snap/docker/common/var-lib-docker/aufs/mnt/5d00bd40adec6662aaec8ea2a5f5ce6a332e9dbfad087a008c5c89b7cac4c22f/usr/bin/share-mnt --stage2 /var/lib/kubelet /var/lib/rancher -- norun]"
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
The text was updated successfully, but these errors were encountered: