Installing kubeadm should mention about kubeadm config file

[inductor]

This is a Feature Request

The Installing kubeadm document says as follows

When using Docker, kubeadm will automatically detect the cgroup driver for the kubelet and set it in the /var/lib/kubelet/config.yaml file during runtime.

If you are using a different CRI, you have to modify the file with your cgroupDriver value, like so:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: <value>

This sounds like saving the yaml file in /var/lib/kubelet/config.yaml IS used in kubeadm init but it actually isn't.

You need to specify the --config option when running kubeadm init

Ref. kubernetes/kubeadm#2240

What would you like to be added

Add kubeadm config and pass it to kubeadm init command.

Example:

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: "systemd"

Why is this needed

To avoid confusion

Comments

[neolit123]

@inductor the kubeadm init command reference page has this block:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file

it links to:
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2

on the godoc page there is an example like so:

...
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# kubelet specific options here
---
...

[neolit123]

To avoid confusion

with the above documentation links / examples i have not seen any recent reports of users being confused.

/priority awaiting-more-evidence
/sig cluster-lifecycle

[inductor]

@neolit123
I get your point but this page should be one-stop place where you can find all the basic information enough to create a single cluster with the given condition; this case it mentions about when using different container runtime, so it should be more clear about that.

My point is that even adding reference link to the config option would be better.

When using Docker, kubeadm will automatically detect the cgroup driver for the kubelet and set it in the /var/lib/kubelet/config.yaml file during runtime.

- If you are using a different CRI, you have to modify the file with your cgroupDriver value, like so:
+ If you are using a different CRI, you have to modify the initial setting with your cgroupDriver value and add pass it when you execute kubeadm init. For further details, please read [Using kubeadm init with a configuration file](/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file)

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: <value>

What do you think?

[neolit123]

proposing a minor change to the above:

If you are using a different CRI, you must pass your cgroupDriver value to kubeadm init, like so:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: <value>

For further details, please read Using kubeadm init with a configuration file.

please send a PR for this if you can.

[inductor]

Created it!

[new-mikha]

To avoid confusion

with the above documentation links / examples i have not seen any recent reports of users being confused.

/priority awaiting-more-evidence
/sig cluster-lifecycle

Hi @neolit123 , on above - ok there is at least one user of that kind now, myself.

See my confusion here:
https://stackoverflow.com/questions/66384374/k8s-1-20-containerd-kubeadm-init-networkpluginnotready

I was totally sure that for just adding systemd as cgroup driver (and nothing more different to the default kubeadm init), this page tells me to print the default config into a file, add the three lines there required for systemd, and supply it to kubeadm.

As you can imagine, I was quite puzzled by the result... Fortunately, I found this page, and realised what a silly sausage I was. The initial suggestion by @inductor , with his config snippet, really helped (thanks!!) - that was exactly what I needed.

Since the recommended runtime now is containerd, and most recommended way is to use systemd as a cgroup driver for kubelet, I think his suggestion is very right and this snippet should be mentioned right at the page that tells about how to install containerd, and its footnote telling about systemd. Honestly, it's really confusing without it.

[neolit123]

hi, we have a new WIP page that will be dedicated to cgroup drivers entirely:
#26786

[aaabdallah]

To avoid confusion

with the above documentation links / examples i have not seen any recent reports of users being confused.
/priority awaiting-more-evidence
/sig cluster-lifecycle

Hi @neolit123 , on above - ok there is at least one user of that kind now, myself.

See my confusion here:
https://stackoverflow.com/questions/66384374/k8s-1-20-containerd-kubeadm-init-networkpluginnotready

I was totally sure that for just adding systemd as cgroup driver (and nothing more different to the default kubeadm init), this page tells me to print the default config into a file, add the three lines there required for systemd, and supply it to kubeadm.

As you can imagine, I was quite puzzled by the result... Fortunately, I found this page, and realised what a silly sausage I was. The initial suggestion by @inductor , with his config snippet, really helped (thanks!!) - that was exactly what I needed.

Since the recommended runtime now is containerd, and most recommended way is to use systemd as a cgroup driver for kubelet, I think his suggestion is very right and this snippet should be mentioned right at the page that tells about how to install containerd, and its footnote telling about systemd. Honestly, it's really confusing without it.

Agreed: I have been confused also for the last couple of hours, and searching finally led to this issue.

[tengqm]

Issue like this should not be closed.
/reopen

[k8s-ci-robot]

@tengqm: Reopened this issue.

In response to this:

Issue like this should not be closed.
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@inductor: This issue is currently awaiting triage.

SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rdxmb]

When talking about the config-file for kubeadm init, we should explain this more generally - not specific to the cgroup-driver.

In my case, I need to set the resolvConf: parameter. For about 8 hours I searched for a way to do that with kubelet --config.

Now I found https://github.com/kubernetes/website/pull/26786/files#diff-7790d405e7f6377d39f6c8a2b74485c6f033400172bda1aade326124ddfba1baR58-R63

So I can implicate that this is a gerenal solution. However, the docs should mention this explicitly.

[rdxmb]

Why is this needed

To avoid confusion

This is also very confusing for me. I could not find how to use the kubelet-configuration with kubeadm in this docs. https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2?utm_source=godoc seems to help in this case.

[neolit123]

@rdxmb did you see this section for kubeadm init?
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file

both the kubeadm init and join documentation pages link to the kubeadm API from where you can pick e.g. v1beta2 and it has examples how to pass KubeletConfiguration to kubeadm init:
https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm

from there users need to understand what fields the KubeletConfiguration has.
the KubeletConfiguration is not managed by the kubeadm team - it's located here:
https://pkg.go.dev/k8s.io/kubelet/config/v1beta1

[neolit123]

and to respond to the original issue (again) here the "installing kubeadm" guide now also has:

For further details, please read "Using kubeadm init with a configuration file".

most users understand what they need to do after clicking that link, but some would still get confused.

due to the cgroup driver configuration issue this PR:
#26786
adds new page explicitly for cgroup divers and after that this issue feels resolved to me.

unless users have more specific requests on how to organize the docs?

[rdxmb]

@rdxmb did you see this section for kubeadm init?
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file

yes, but that site does not tell anything about the syntax and structure for that config file.
For me as an system engineer I am not able to create a config file with the information included on that site.

both the kubeadm init and join documentation pages link to the kubeadm API from where you can pick e.g. v1beta2 and it has examples how to pass KubeletConfiguration to kubeadm init:
https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm

I must admit that I have never scrolled to the end of that page. For me, it was not transparent that I have to use "Directories" to get the link to v1beta2, which will finally lead me to the information I wanted. That's why I asked for the link to https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2 itself.

from there users need to understand what fields the KubeletConfiguration has.
the KubeletConfiguration is not managed by the kubeadm team - it's located here:
https://pkg.go.dev/k8s.io/kubelet/config/v1beta1

Here is no link to "Directories". Where do I find the syntax for the KubeletConfiguration within the yaml-config-file?

and to respond to the original issue (again) here the "installing kubeadm" guide now also has:
For further details, please read "Using kubeadm init with a configuration file".
most users understand what they need to do after clicking that link, but some would still get confused.

I was confused because that link does not explain how to use the KubeletConfiguration with kubeadm init --config ...

unless users have more specific requests on how to organize the docs?

This was my suggestion: #22916 (comment) . Another solution would be here: #27192

Maybe someone else can check what will be helpful here.

EDIT:

#kubeadm config print
missing subcommand; "print" is not meant to be run on its own
To see the stack trace of this error execute with --v=5 or higher

Could we also have the 3 or 4 steps to create a configuration file instead of linking to the print command, which is in fact view, pipe them to a file and then run with kubeadm config migrate --old-config ?

[neolit123]

yes, but that site does not tell anything about the syntax and structure for that config file.
For me as an system engineer I am not able to create a config file with the information included on that site.

it's an API presented to the user in a standard godoc format.
structure fields map to JSON / YAML, e.g.:
https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#NodeRegistrationOptions

i'm surprised that you as a system engineer are having trouble parsing this aspect.
what is the exact change you would like to see?

I must admit that I have never scrolled to the end of that page. For me, it was not transparent that I have to use "Directories" to get the link to v1beta2, which will finally lead me to the information I wanted. That's why I asked for the link to https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2 itself.

that's actually a bug. prior to the new godoc we design it was possible to HTML anchor link to the subdirectories.
i've just logged this issue golang/go#45210

Here is no link to "Directories". Where do I find the syntax for the KubeletConfiguration within the yaml-config-file?

it's not a directory because the KubeletConfiguration is not a kubeadm structure, it's a kubelet structure.
kubeadm just allows you pass it to the kubelet, because kubeadm manages a kubelet.

the kubeadm v1beta2 page links to the KubeletConfiguration, did you see this?

See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or https://godoc.org/k8s.io/kubelet/config/v1beta1#KubeletConfiguration for kubelet official documentation.

here https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2

I was confused because that link does not explain how to use the KubeletConfiguration with kubeadm init --config ...

currently users must navigate to https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm or https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2 directly.

a number of kubeadm pages link to that already.

This was my suggestion: #22916 (comment) .

this is a bit vague to me, still.

Another solution would be here: #27192

this change seems fine to me, but i cannot approve it since it's for the kubelet pages.

Maybe someone else can check what will be helpful here.

sure, we can keep this issue open until that happens.
one problem that i have with similar requests is that users want a change, but they are not exactly sure what they want to be changed. the feedback is vague.

Could we also have the 3 or 4 steps to create a configuration file instead of linking to the print command, which is in fact view, pipe them to a file and then run with kubeadm config migrate --old-config ?

its not "config view", but rather "config print init-defaults".
including a placeholder stub seems OK to me in the lines of:

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration

but nothing more, just the link to https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#pkg-subdirectories which is already there.

PRs welcome.

[rdxmb]

@neolit123 First, thank you for taking so much time and your questions to understand that!

I also want to help with my point of view so I hope this will be transparent. First, I want to answer your points. Then, in my next comment, I'd like to describe my "jouney" through the docs on the wrong tracks...

https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2 itself.

that's actually a bug. prior to the new godoc we design it was possible to HTML anchor link to the subdirectories.
i've just logged this issue golang/go#45210

I've just commented it...

Here is no link to "Directories". Where do I find the syntax for the KubeletConfiguration within the yaml-config-file?

it's not a directory because the KubeletConfiguration is not a kubeadm structure, it's a kubelet structure.
kubeadm just allows you pass it to the kubelet, because kubeadm manages a kubelet.

Ok, thanks. You see, this is the "inner" way to see it. For administrating it, it is not that easy to see why it's so complicated to find those simple things (see also my next comment)

the kubeadm v1beta2 page links to the KubeletConfiguration, did you see this?

To be honest: No. But after so many hours lost in space, I did not take the time to have a look at that in detail. So this definitely was my fault. Sorry.

`See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

I already knew that I have to use the config file instead of the command-line-tools. So I know this page, but do not read it for my actual problem. (see next comment: lost in space)

https://godoc.org/k8s.io/kubelet/config/v1beta1#KubeletConfiguration for kubelet official documentation.`

This did not seem to be a syntax for me which can be used for my kubelet configuration.

here https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2

Yes, this is perfect! This is what I was looking for for 2 days :)

I was confused because that link does not explain how to use the KubeletConfiguration with kubeadm init --config ...

currently users must navigate to https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm or https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2 directly.

This seems to be the biggest barrier here.

a number of kubeadm pages link to that already.

Do you find one of them when searching for kubelet config :) ?

This was my suggestion: #22916 (comment) .

this is a bit vague to me, still.

I will add a new comment or maybe another issue to discuss that.

Another solution would be here: #27192

this change seems fine to me, but i cannot approve it since it's for the kubelet pages.

Thanks.

one problem that i have with similar requests is that users want a change, but they are not exactly sure what they want to be changed. the feedback is vague.

I see.

Ok, I will try to help more precisely. The thing for me is to understand the way it works without having the docs I would have needed - and the give the feedback what would have been helpful.

its not "config view", but rather "config print init-defaults".

I would never have the idea that I have to do that to create the config I need to change, when I have to change the kubelet-config.

including a placeholder stub seems OK to me in the lines of:

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration

Yes, this exactly should be explained on a page like https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file

but nothing more, just the link to https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#pkg-subdirectories which is already there.

and broken ;)

PRs welcome.

Will you make one to fix the link to https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm?utm_source=godoc#section-directories ? If not, I can do that.

[rdxmb]

yes, but that site does not tell anything about the syntax and structure for that config file.
For me as an system engineer I am not able to create a config file with the information included on that site.

This refered to https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file , not to the API documentation.

i'm surprised that you as a system engineer are having trouble parsing this aspect.
what is the exact change you would like to see?

I'll try to show you my confusing way through the docs. I hope we see wenn I got lost :)

1. I have 3 small clusters running created with kubeadm. For the provisioning I use https://github.com/geerlingguy/ansible-role-kubernetes . I know that we have to switch to a kubelet-config-file instead of the command line because of deprecation for most arguments. I am starting with zero knowledge about this config-file.

2. I am starting with https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ . I have no idea that what I have to do has anything to do with kubadm. I read the example for the config file and know that I have to start kubelet with --config and this file. EDIT: Fixed with Update kubelet-config-file.md #27192

This is the first time I get lost - where I created this issue: #27188 . So you see I was on the very wrong way.

3. Another confusing thing is here:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node

If you are using a different CRI, you must pass your cgroupDriver value to kubeadm init, like so:

For now I am still using docker, so I have no idea that this is also the solution for my case.

use the KubeletConfiguration instead (stored in /var/lib/kubelet/config.yaml by default).

BUT: editing /var/lib/kubelet/config.yaml is not a good idea... . I know that this is NOT the solution. Lost in space...

4. Link to https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file

EDIT: hopefully better with #27201

It's possible to configure kubeadm init with a configuration file instead of command line flags, and some more advanced features may only be available as configuration file options. This file is passed using the --config flag and it must contain a ClusterConfiguration structure and optionally more structures separated by ---\n Mixing --config with others flags may not be allowed in some cases.

This really does NOT sound that it could be the solution for my problem. However, if I had a full example of the config file meant here, I would have seen that it also had a KubeletConfiguration-part. So I had no idea why I should print my kubeadm config and migrate it ...

5. The link with the list https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm?utm_source=godoc#pkg-subdirectories is broken, we talked about that... I am still lost ... EDIT: fixed Update kubeadm-init.md #27199

6. Searching the web again I finally find kubeadm overwrites /var/lib/kubelet/config.yaml even if you already have it kubeadm#2240 and the link to this issue.... and get the idea that this could be a way for me, although changing the cgroup-Driver is not my problem... (Thanks to your comments there! and the PR of @inductor!)

7. Here I am :)

[neolit123]

@neolit123 First, thank you for taking so much time and your questions to understand that!

no problem, but i don't have much time to debate this close to a kubernetes release (1.21).
your feedback seems like a story in the lines of "hey, i saw a bunch of problems in your docs".
if you have any suggestions just log separate issues / send PRs.

EOD, every single change gets approved by the kubernetes/website maintainers and not by component maintainers like me.

Will you make one to fix the link to https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm?utm_source=godoc#section-directories ? If not, I can do that.

you can update the link in k/website.
i will try to fix the general pkg.go.dev problem at golang/go#45210

[rdxmb]

EOD, every single change gets approved by the kubernetes/website maintainers and not by component maintainers like me.

Oh, I thought you were a docs maintainer. Thanks for your help. still.

you can update the link in k/website.

#27199

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[rdxmb]

/close

[k8s-ci-robot]

@rdxmb: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.