Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubeadm: add separate page for configuring / migrating cgroup driver #26786

Merged
merged 1 commit into from
Mar 30, 2021
Merged

kubeadm: add separate page for configuring / migrating cgroup driver #26786

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion content/en/docs/setup/production-environment/container-runtimes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Changing the settings such that your container runtime and kubelet use `systemd`
stabilized the system. To configure this for Docker, set `native.cgroupdriver=systemd`.

{{< caution >}}
Changing the cgroup driver of a Node that has joined a cluster is strongly *not* recommended.
Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation.
If the kubelet has created Pods using the semantics of one cgroup driver, changing the container
runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox
for such existing Pods. Restarting the kubelet may not solve such errors.
Expand All @@ -57,6 +57,11 @@ If you have automation that makes it feasible, replace the node with another usi
configuration, or reinstall it using automation.
{{< /caution >}}

### Migrating to the `systemd` driver in kubeadm managed clusters

Follow this [Migration guide](/tasks/administer-cluster/kubeadm/configure-cgroup-driver)
if you wish to migrate to the `systemd` cgroup driver in existing kubeadm managed clusters.

## Container runtimes

{{% thirdparty-content %}}
Expand Down
34 changes: 9 additions & 25 deletions content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ sudo mkdir -p /opt/cni/bin
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | sudo tar -C /opt/cni/bin -xz
```

Define the directory to download command files
Define the directory to download command files

{{< note >}}
The `DOWNLOAD_DIR` variable must be set to a writable directory.
Expand Down Expand Up @@ -284,33 +284,17 @@ See the [Kubeadm Troubleshooting guide](/docs/setup/production-environment/tools
The kubelet is now restarting every few seconds, as it waits in a crashloop for
kubeadm to tell it what to do.

## Configure cgroup driver used by kubelet on control-plane node
## Configuring a cgroup driver

When using Docker, kubeadm will automatically detect the cgroup driver for the kubelet
and set it in the `/var/lib/kubelet/config.yaml` file during runtime.
Both the container runtime and the kubelet have a property called
["cgroup driver"](/docs/setup/production-environment/container-runtimes/), which is important
for the management of cgroups on Linux machines.

If you are using a different CRI, you must pass your `cgroupDriver` value to `kubeadm init`, like so:

```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: <value>
```

For further details, please read [Using kubeadm init with a configuration file](/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file).

Please mind, that you **only** have to do that if the cgroup driver of your CRI
is not `cgroupfs`, because that is the default value in the kubelet already.

{{< note >}}
Since `--cgroup-driver` flag has been deprecated by the kubelet, if you have that in `/var/lib/kubelet/kubeadm-flags.env`
or `/etc/default/kubelet`(`/etc/sysconfig/kubelet` for RPMs), please remove it and use the KubeletConfiguration instead
(stored in `/var/lib/kubelet/config.yaml` by default).
{{< /note >}}

The automatic detection of cgroup driver for other container runtimes
like CRI-O and containerd is work in progress.
{{< warning >}}
Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail.
{{< warning >}}

See [Configuring a cgroup driver](/tasks/administer-cluster/kubeadm/configure-cgroup-driver) for more details.

## Troubleshooting

Expand Down
127 changes: 127 additions & 0 deletions content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
---
title: Configuring a cgroup driver
content_type: task
weight: 10
---

<!-- overview -->

This page explains how to configure the kubelet cgroup driver to match the container
runtime cgroup driver for kubeadm clusters.

## {{% heading "prerequisites" %}}

You should be familiar with the Kubernetes
[container runtime requirements](/docs/setup/production-environment/container-runtimes).

<!-- steps -->

## Configuring the container runtime cgroup driver

The [Container runtimes](/docs/setup/production-environment/container-runtimes) page
explains that the `systemd` driver is recommended for kubeadm based setups instead
of the `cgroupfs` driver, because kubeadm manages the kubelet as a systemd service.

The page also provides details on how to setup a number of different container runtimes with the
`systemd` driver by default.

## Configuring the kubelet cgroup driver

kubeadm allows you to pass a `KubeletConfiguration` structure during `kubeadm init`.
This `KubeletConfiguration` can include the `cgroupDriver` field which controls the cgroup
driver of the kubelet.

{{< note >}}

{{< feature-state for_k8s_version="v1.21" state="stable" >}}

If the user is not setting the `cgroupDriver` field under `KubeletConfiguration`,
`kubeadm init` will default it to `systemd`.
{{< /note >}}

A minimal example of configuring the field explicitly:

```yaml
# kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
kubernetesVersion: v1.21
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2?utm_source=godoc uses

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    cgroup-driver: "cgroupfs"

to define the cgroupfs.

Confusing...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use kubeletExtraArgs, because the cgroup-driver flag is deprecated.
this new page has the right config example.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand. However, this is very confusing, so where is an actual full reference for the kubeadm-kubelet-config-file to configure everything in one file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are also the support forums and #kubeadm on the k8s slack for users that need help:
https://github.com/kubernetes/kubernetes/blob/master/SUPPORT.md

i will send a PR to remove the kubeletExtraArgs examples in https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2?utm_source=godoc

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are also the support forums and #kubeadm on the k8s slack for users that need help:
https://github.com/kubernetes/kubernetes/blob/master/SUPPORT.md

I know. But the docs should be the first place to search, shouldn't it?

i will send a PR to remove the kubeletExtraArgs examples in https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2?utm_source=godoc

👍

```

Such a configuration file can then be passed to the kubeadm command:

```shell
kubeadm init --config kubeadm-config.yaml
```

{{< note >}}
Kubeadm uses the same `KubeletConfiguration` for all nodes in the cluster.
The `KubeletConfiguration` is stored in a [ConfigMap](docs/concepts/configuration/configmap)
neolit123 marked this conversation as resolved.
Show resolved Hide resolved
object under the `kube-system` namespace.
neolit123 marked this conversation as resolved.
Show resolved Hide resolved

Executing the sub commands `init`, `join` and `upgrade` would result in kubeadm
writing the `KubeletConfiguration` as a file under `/var/lib/kubelet/config.yaml`
and passing it to the local node kubelet.
{{< /note >}}

## Using the `cgroupfs` driver

As this guide explains using the `cgroupfs` driver with kubeadm is not recommended.

To continue using `cgroupfs` and to prevent `kubeadm upgrade` from modifying the
`KubeletConfiguration` cgroup driver on existing setups, you must be explicit
about its value. This applies to a case where you do not wish future versions
of kubeadm to apply the `systemd` driver by default.

See the below section on "Modify the kubelet ConfigMap" for details on
how to be explicit about the value.

If you wish to configure a container runtime to use the `cgroupfs` driver,
you must refer to the documentation of the container runtime of your choice.

## Migrating to the `systemd` driver

To change the cgroup driver of an existing kubeadm cluster to `systemd` in-place,
a similar procedure to a kubelet upgrade is required. This must include both
steps outlined below.

{{< note >}}
Alternatively, it is possible to replace the old nodes in the cluster with new ones
that use the `systemd` driver. This requires executing only the first step below
before joining the new nodes and ensuring the workloads can safely move to the new
nodes before deleting the old nodes.
{{< /note >}}

### Modify the kubelet ConfigMap

- Find the kubelet ConfigMap name using `kubectl get cm -n kube-system | grep kubelet-config`.
- Call `kubectl edit cm kubelet-config-x.yy -n kube-system` (replace `x.yy` with
the Kubernetes version).
- Either modify the existing `cgroupDriver` value or add a new field that looks like this:

```yaml
cgroupDriver: systemd
```
This field must be present under the `kubelet:` section of the ConfigMap.

### Update the cgroup driver on all nodes

For each node in the cluster:

- [Drain the node](/docs/tasks/administer-cluster/safely-drain-node) using `kubectl drain <node-name> --ignore-daemonsets`
- Stop the kubelet using `systemctl stop kubelet`
- Stop the container runtime
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes, we need to clean up the container runtime dir.
Not sure if this is needed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from my tests this was not needed.
we can update the guide after this PR merges if we find gaps.

- Modify the container runtime cgroup driver to `systemd`
- Set `cgroupDriver: systemd` in `/var/lib/kubelet/config.yaml`
- Start the container runtime
- Start the kubelet using `systemctl start kubelet`
- [Uncordon the node](/docs/tasks/administer-cluster/safely-drain-node) using `kubectl uncordon <node-name>`

Execute these steps on nodes one at a time to ensure workloads
have sufficient time to schedule on different nodes.

Once the process is complete ensure that all nodes and workloads are healthy.