Repository navigation
Kubeadm operator #2505
Description
Enhancement Description
- One-line enhancement description (can be used as a release note): Kubeadm operator
- Kubernetes Enhancement Proposal:
Summary
Kubeadm operator would like to enable declarative control of kubeadm workflows, automating the execution and the
orchestration of such tasks across existing nodes in a cluster.
Motivation
Kubeadm binary can execute operations only on the machine where it is running e.g. it is not possible to execute
operations on other nodes, to copy files across nodes, etc.
As a consequence, most of the kubeadm workflows, like kubeadm upgrade,
consists of a complex sequence of tasks that should be manually executed and orchestrated across all the existing nodes
in the cluster.
Such a user experience is not ideal due to the error-prone nature of humans running commands. The manual approach
can be considered a blocker for implementing more complex workflows such as rotating certificate authorities,
modifying the settings of an existing cluster or any task that requires coordination of more than one Kubernetes node.
This KEP aims to address such problems by applying the operator pattern to kubeadm workflows.
- Discussion Link:
- Primary contact (assignee): @fabriziopandini
- Responsible SIGs: sig-cluster-lifecycle
- Enhancement target (which target equals to which milestone):
- Alpha release target (x.y): TODO
- Beta release target (x.y): TODO
- Stable release target (x.y): TODO
- Alpha
- KEP (
k/enhancements) update PR(s): - Code (
k/k) update PR(s): - Docs (
k/website) update PR(s):
- KEP (
Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.
Activity
fejta-bot commentedon Jun 9, 2021
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
neolit123 commentedon Jun 9, 2021
/remove-lifecycle stale
k8s-triage-robot commentedon Sep 7, 2021
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle stale/lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
neolit123 commentedon Sep 8, 2021
/remove-lifecycle stale
pacoxu commentedon Sep 14, 2021
Any plan for this feature
neolit123 commentedon Sep 14, 2021
fabriziopandini commentedon Sep 15, 2021
I have a prototype and many ideas around this and the kubeadm, library...
2 remaining items
fabriziopandini commentedon Dec 14, 2021
/lifecycle frozen
chendave commentedon Feb 10, 2022
/cc
I will evaluate this feature and see if we can move forward this.
pacoxu commentedon Jun 6, 2022
DaoCloud-OpenSource/kubeadm-operator#53 I did some init implementation for
upgrade and certs renew.I got a problem to implement the kubeadm operator.
systemctl restart kubeletinside a pod for security concerns.If kubelet is not restarted, the apiserver will be the new version and kubelet will be n-1 version.
/usr/bin/kubeletand/usr/bin/kubelet-neware different. If they are different, it will trigger to executesystemctl stop kubelet && /usr/bin/cp /usr/bin/kubelet-new /usr/bin/kubelet && systemctl restart kubelet. This is not that convenient.Some other thoughts, can we kill kubelet inside pod? Or can we add a flag file in hostPath to let kubelet know it should be restarted? I don't know if there is a simple way to restart kubelet by a kubelet API calling or other methods.
chendave commentedon Jun 6, 2022
@pacoxu I am also working on this, all those work for my side are still under internal maintained. It's okay for you to join in but we'd better to sync with each other to avoid the duplicated effort on this.
our work is based on @fabriziopandini's POC, how about yours?
This is indeed an issue, so far it is fine since kubelet 1.24 will continue to work with apisever 1.25, but this must be taken care because of the changing from cri support on docker.
pacoxu commentedon Jun 6, 2022
DaoCloud-OpenSource/kubeadm-operator#2 The same. The POC was removed from kubeadm code base in kubernetes/kubeadm#2342.
chendave commentedon Jun 6, 2022
cc @ruquanzhao for awareness, we need to figure out a solution on the upgrade of
kubeletas well.neolit123 commentedon Jun 6, 2022
big +1 for multiple people collaborating on this. happy to help with more ideas / review when needed.
i don't think there is an API to restart kubelet.
then again, that would restart a particular kubelet and we want to upgrade the binary too.
this sounds like one of the ways to do it. i don't think i have and significantly better ideas right now.
e.g. we could (somehow) deploy scripts on the hosts that manage the kubelet restart cycle and upgrade even if the pods that the operator DS deploys are killed due to a kubelet restart.
the operator will have have to have super powers on the hosts, so it will be considered as a trusted "actor"...there is no other way to manage component upgrade (kubeadm, kubelet) and cert rotation, etc.
pacoxu commentedon Jun 6, 2022
I can build the workaround into a script.
If we restart kubelet inside pod, I tried to mount
/run/systemd&/var/run/dbus/system_bus_socketand/sys/fs/cgroupto the agent and set privileged/hostPID to true, but thesystemctl restart kubeletstill failed. I am not sure what I am missing.pacoxu commentedon Jun 17, 2022
I write a simple kubelet-reloader
/usr/bin/kubelet-new./usr/bin/kubeletand restart kubelet.Currently the kubeadm-operator v0.1.0 can support upgrade cross versions like v1.22 to v1.24.
kubectl/kubelet/kubeadmand upgrade./usr/bin/kubelet-newfor kubelet reloader.See quick-start.
neolit123 commentedon Jun 17, 2022
Thats great. I think we should have our discussion on the k/kubeadm issue to have it in one place. Also cross coordinate with @chendave to avoid duplicated work.
pacoxu commentedon Jun 30, 2022
kubernetes/kubeadm#2317 may be the right place.
chendave commentedon Apr 20, 2023
FYI - We have all the initial scope of the KEP #1239 implemented here: https://github.com/chendave/kubeadm-operator
But it is still a just POC.
@pacoxu @neolit123 @ruquanzhao
pacoxu commentedon May 10, 2023
We have a try to ask if kubeadm operator can be a sig-clusterlifecyle subproject earlier this year. Some context can be found in https://docs.google.com/document/d/1Gmc7LyCIL_148a9Tft7pdhdee0NBHdOfHS1SAF0duI4/edit#heading=h.xm2jvfwtcfuz sig clusterlifecycle weekly meeting and kubernetes-sigs/cluster-api#7044 cluster-api gathering feedback issue.