Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs" #18776

Closed
robdaems opened this issue Feb 28, 2018 · 9 comments
Assignees
Labels

Comments

@robdaems
Copy link

Version
[root@rhel ~]# openshift version
openshift v3.7.1+ab0f056
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8
[root@rhel ~]# uname -a
Linux rhel 3.10.0-693.17.1.el7.x86_64 #1 SMP Sun Jan 14 10:36:03 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@rhel ~]# docker --version
Docker version 17.12.1-ce, build 7390fc6
[root@rhel ~]#
Steps To Reproduce

On a Clean RHEL 7.4 server (1 machine) installation I try to run openshift as a All-in-One

[root@rhel ~]# openshift start

Googling around doesn't give me a proper solution.

Current Result
W0228 16:22:37.679295    2019 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
I0228 16:22:37.753317    2019 docker_service.go:209] Docker cri networking managed by kubernetes.io/no-op
F0228 16:22:37.761061    2019 node.go:264] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"
@jwforres
Copy link
Member

@openshift/sig-pod

@aveshagarwal
Copy link
Contributor

This seems just a configuration issue, change docker's cgroup driver to systemd.

@aveshagarwal
Copy link
Contributor

use with docker: --exec-opt native.cgroupdriver=systemd

@robdaems
Copy link
Author

robdaems commented Mar 1, 2018

@aveshagarwal

use with docker: --exec-opt native.cgroupdriver=systemd

Which file(s) do I've to change and which variable?
Since I found this solution also on google, but it didn't help. So I started again with a clean install.

[root@rhel ~]# docker info | grep Cgroup
Cgroup Driver: cgroupfs
[root@rhel ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2018-03-01 12:31:28 CET; 1h 39min ago
     Docs: https://docs.docker.com
 Main PID: 18306 (dockerd)
   Memory: 25.9M
   CGroup: /system.slice/docker.service
           ├─18306 /usr/bin/dockerd
           └─18309 docker-containerd --config /var/run/docker/containerd/containerd.toml

Mar 01 12:31:27 rhel dockerd[18306]: time="2018-03-01T12:31:27+01:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
Mar 01 12:31:27 rhel dockerd[18306]: time="2018-03-01T12:31:27+01:00" level=info msg="containerd successfully booted in 0.007546s" module=containerd
Mar 01 12:31:27 rhel dockerd[18306]: time="2018-03-01T12:31:27.832225213+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Mar 01 12:31:27 rhel dockerd[18306]: time="2018-03-01T12:31:27.832986792+01:00" level=info msg="Loading containers: start."
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.068006399+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.246296496+01:00" level=info msg="Loading containers: done."
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.272793007+01:00" level=info msg="Docker daemon" commit=7390fc6 graphdriver(s)=overlay2 version=17.12.1-ce
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.273026469+01:00" level=info msg="Daemon has completed initialization"
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.298266653+01:00" level=info msg="API listen on /var/run/docker.sock"
Mar 01 12:31:28 rhel systemd[1]: Started Docker Application Container Engine.

@robdaems
Copy link
Author

robdaems commented Mar 1, 2018

It's fixed by adapting the line in /usr/lib/systemd/system/docker.service

ExecStart=/usr/bin/dockerd \
          --exec-opt native.cgroupdriver=systemd

@krumware
Copy link

Seeing this issue in WSL.
We don't have easy control over systemd, so the above instructions are difficult to follow.

Users may also be able to add the instructions in the file /etc/default/docker.
Adding the line: DOCKER_OPTS="--exec-opt native.cgroupdriver=systemd"

After doing this, our new error is

F0627 12:04:35.582876    3520 network.go:179] error: Could not initialize Kubernetes Proxy. You must run this process as root (and if containerized, in the host network namespace as privileged) to use the service proxy: failed to initialize iptables: error appending rule: exit status 1: iptables: No chain/target/match by that name.

@harshalkwagh
Copy link

in my case i already have --exec-opt native.cgroupdriver=systemd option in /usr/lib/systemd/system/docker.service file but still my CgroupDriver is using cgroupfs

ExecStart=/usr/bin/dockerd-current
--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current
--default-runtime=docker-runc
--exec-opt native.cgroupdriver=systemd
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current
--init-path=/usr/libexec/docker/docker-init-current
--seccomp-profile=/etc/docker/seccomp.json \

Docker info:
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 9c3c5f853ebf0ffac0d087e94daef462133b69c7 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)

How to change cgroup to systemd

@ghost
Copy link

ghost commented Sep 12, 2019

--exec-opt native.cgroupdriver=systemd

location for docker 17.03 on ubuntu 16.04 = /lib/systemd/system/docker.service

@sirsikar-lalit
Copy link

I was able to resolve this issue for my use-case by having the same cgroup driver for docker and kubelet. In my case on CentOS 7.6 I could fix the issue by adding --exec-opt native.cgroupdriver=systemd to docker systemd process and adding --cgroup-driver=systemd to kubelet systemd process. This way both kubelet and docker are consuming the same cgroup-driver and both operate normally.

Most likely these drivers can be set with any other driver types as well but that was not a part of my testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

8 participants