-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cloud-controller-manager exited #7328
Comments
It does, actually, although it only checks for our custom RBAC, not core Kubernetes RBAC. k3s/pkg/daemons/control/server.go Lines 380 to 392 in 944f811
Can you post the complete server logs? In the snippet you quoted I see some slow SQL warnings:
This makes me think that there's something else going on with your datastore performance that's causing some of the core RBAC to be missing at this point even though our custom RBAC is present. Access to the What kind of disk are you using? How many cores and how much memory does this node have? It looks like you're just using the embedded sqlite datastore, which is generally pretty forgiving of slow disk, but the times between your log messages are much longer than I would expect. |
Currently, I m not able to attach full logs from journalctl, as the error is once appearing and once not. |
If you're running on ec2, have you checked the cloudwatch metrics to see if your instance or disk volumes are running out of CPU or io credits and being throttled? |
Yeah something is definitely wrong with your disk. Just creating the very first row in the database during initial setup caused a slow SQL error. This process normally takes like, milliseconds at the most.
1.5 seconds to insert the very first row in the databse, before ANYTHING else is running, is bonkers slow. Figure out why the IO on this node is so bad and your problems should go away. |
Environmental Info:
K3s Version:
Node(s) CPU architecture, OS, and Version:
AWS EC2 instance:
Cluster Configuration: 1 server only
Describe the bug:
cloud-controller-manager exited due to
Apr 20 13:02:35 <hidden> k3s[84782]: time="2023-04-20T13:02:35+02:00" level=error msg="cloud-controller-manager exited: unable to load configmap based request-header-client-ca-file: configmaps \"extension-apiserver-authentication\" is forbidden: User \"k3s-cloud-controller-manager\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""
, the role was created at"2023-04-20T13:02:30Z"
and the role binding was created at"2023-04-20T13:02:36Z"
. It means that at the moment when cloud-controller-manager tries to get config map, it has not permissions due to lack of role binding. And it causes that taintnode.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
remains on the node and pods stay inpending
state.Restarting the k3s service resolves the issue, cloud-controller is restarted and the role binding already exists, that is probably why it is working after k3s restart, but it is not how it should work.
Steps To Reproduce:
I do not have steps to reproduce, as it occurs periodically.
Expected behavior:
cloud-controller-manager running properly, maybe it should ensure that role and role binding are created before trying to get the configmap.
Actual behavior:
cloud-controller-manager exited, it was not able to get the required config map due to lack of permissions.
Additional context / logs:
role details:
role binding details:
journalctl logs related to cloud controller:
The text was updated successfully, but these errors were encountered: