openshift/origin docker container fails to start: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs" #14766

queer · 2017-06-20T08:21:44Z

Attempting to run the Origin docker container using the command listed in the docs fails with the error in the title.

Version

N/A, haven't even gotten far enough for it to work

Steps To Reproduce

Run the command listed on the docs page:

sudo docker run -d --name "origin" \
    --privileged --pid=host --net=host \
    -v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys -v /sys/fs/cgroup:/sys/fs/cgroup:rw \
    -v /var/lib/docker:/var/lib/docker:rw \
    openshift/origin:latest start

docker logs -f origin and wait for output to finish

Current Result

F0620 08:05:16.627956    6766 node.go:297] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

Expected Result

Container should start up without issues

Additional Information

lsb_release -a:

user@host ~> lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 8.8 (jessie)
Release:	8.8
Codename:	jessie

docker -v:

user@host ~> docker -v
Docker version 17.03.1-ce, build c6d412e

Full container logs:

user@host ~> docker logs -f origin
W0620 08:04:59.197432    6766 start_master.go:291] Warning: assetConfig.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console, master start will continue.
W0620 08:04:59.197807    6766 start_master.go:291] Warning: assetConfig.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console, master start will continue.
W0620 08:04:59.197840    6766 start_master.go:291] Warning: auditConfig.auditFilePath: Required value: audit can not be logged to a separate file, master start will continue.
I0620 08:04:59.213516    6766 plugins.go:101] No cloud provider specified.
2017-06-20 08:04:59.224894 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp AAA.BBB.CCC.DDD:4001: getsockopt: connection refused"; Reconnecting to {AAA.BBB.CCC.DDD:4001 <nil>}
2017-06-20 08:04:59.225081 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp AAA.BBB.CCC.DDD:4001: getsockopt: connection refused"; Reconnecting to {AAA.BBB.CCC.DDD:4001 <nil>}
2017-06-20 08:04:59.225164 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp AAA.BBB.CCC.DDD:4001: getsockopt: connection refused"; Reconnecting to {AAA.BBB.CCC.DDD:4001 <nil>}
2017-06-20 08:04:59.225234 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp AAA.BBB.CCC.DDD:4001: getsockopt: connection refused"; Reconnecting to {AAA.BBB.CCC.DDD:4001 <nil>}
2017-06-20 08:04:59.244676 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp AAA.BBB.CCC.DDD:4001: getsockopt: connection refused"; Reconnecting to {AAA.BBB.CCC.DDD:4001 <nil>}
I0620 08:04:59.578567    6766 start_master.go:430] Starting master on 0.0.0.0:8443 (v3.6.0-alpha.2+2d90d52-359)
I0620 08:04:59.578670    6766 start_master.go:431] Public master address is https://AAA.BBB.CCC.DDD:8443
I0620 08:04:59.578728    6766 start_master.go:435] Using images from "openshift/origin-<component>:v3.6.0-alpha.2"
2017-06-20 08:04:59.578989 I | embed: peerTLS: cert = openshift.local.config/master/etcd.server.crt, key = openshift.local.config/master/etcd.server.key, ca = openshift.local.config/master/ca.crt, trusted-ca = , client-cert-auth = true
2017-06-20 08:04:59.579788 I | embed: listening for peers on https://0.0.0.0:7001
2017-06-20 08:04:59.579874 I | embed: listening for client requests on 0.0.0.0:4001
2017-06-20 08:04:59.593344 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp AAA.BBB.CCC.DDD:4001: getsockopt: connection refused"; Reconnecting to {AAA.BBB.CCC.DDD:4001 <nil>}
2017-06-20 08:04:59.599489 I | etcdserver: name = openshift.local
2017-06-20 08:04:59.599573 I | etcdserver: data dir = openshift.local.etcd
2017-06-20 08:04:59.599595 I | etcdserver: member dir = openshift.local.etcd/member
2017-06-20 08:04:59.599612 I | etcdserver: heartbeat = 100ms
2017-06-20 08:04:59.599642 I | etcdserver: election = 1000ms
2017-06-20 08:04:59.599657 I | etcdserver: snapshot count = 100000
2017-06-20 08:04:59.599681 I | etcdserver: advertise client URLs = https://AAA.BBB.CCC.DDD:4001
2017-06-20 08:04:59.599714 I | etcdserver: initial advertise peer URLs = https://AAA.BBB.CCC.DDD:7001
2017-06-20 08:04:59.599739 I | etcdserver: initial cluster = openshift.local=https://AAA.BBB.CCC.DDD:7001
2017-06-20 08:04:59.604307 I | etcdserver: starting member d2ca0e9a4ad0cca7 in cluster dda1658ef20fdb45
2017-06-20 08:04:59.604538 I | raft: d2ca0e9a4ad0cca7 became follower at term 0
2017-06-20 08:04:59.604602 I | raft: newRaft d2ca0e9a4ad0cca7 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2017-06-20 08:04:59.604678 I | raft: d2ca0e9a4ad0cca7 became follower at term 1
2017-06-20 08:04:59.610276 W | auth: simple token is not cryptographically signed
2017-06-20 08:04:59.622067 I | etcdserver: starting server... [version: 3.2.0, cluster version: to_be_decided]
2017-06-20 08:04:59.622182 I | embed: ClientTLS: cert = openshift.local.config/master/etcd.server.crt, key = openshift.local.config/master/etcd.server.key, ca = openshift.local.config/master/ca.crt, trusted-ca = , client-cert-auth = true
2017-06-20 08:04:59.643352 I | etcdserver/membership: added member d2ca0e9a4ad0cca7 [https://AAA.BBB.CCC.DDD:7001] to cluster dda1658ef20fdb45
2017-06-20 08:05:00.605265 I | raft: d2ca0e9a4ad0cca7 is starting a new election at term 1
2017-06-20 08:05:00.605394 I | raft: d2ca0e9a4ad0cca7 became candidate at term 2
2017-06-20 08:05:00.605466 I | raft: d2ca0e9a4ad0cca7 received MsgVoteResp from d2ca0e9a4ad0cca7 at term 2
2017-06-20 08:05:00.605492 I | raft: d2ca0e9a4ad0cca7 became leader at term 2
2017-06-20 08:05:00.605500 I | raft: raft.node: d2ca0e9a4ad0cca7 elected leader d2ca0e9a4ad0cca7 at term 2
2017-06-20 08:05:00.606065 I | etcdserver: setting up the initial cluster version to 3.2
2017-06-20 08:05:00.606148 I | etcdserver: published {Name:openshift.local ClientURLs:[https://AAA.BBB.CCC.DDD:4001]} to cluster dda1658ef20fdb45
2017-06-20 08:05:00.606161 I | embed: ready to serve client requests
2017-06-20 08:05:00.606843 I | embed: serving client requests on [::]:4001
2017-06-20 08:05:00.662051 N | etcdserver/membership: set the initial cluster version to 3.2
2017-06-20 08:05:00.662431 I | etcdserver/api: enabled capabilities for version 3.2
I0620 08:05:00.699501    6766 run.go:85] Started etcd at AAA.BBB.CCC.DDD:4001
2017-06-20 08:05:00.803943 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-06-20 08:05:00.804268 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-06-20 08:05:00.804356 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-06-20 08:05:00.804462 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-06-20 08:05:00.807063 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-06-20 08:05:00.807279 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-06-20 08:05:00.811270 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-06-20 08:05:00.811361 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I0620 08:05:00.818752    6766 run_components.go:113] Using default project node label selector: 
I0620 08:05:00.822754    6766 master.go:169] Starting OAuth2 API at /oauth
I0620 08:05:00.822815    6766 master.go:177] Starting Web Console /console/
I0620 08:05:00.826591    6766 clusterquotamapping.go:101] Starting ClusterQuotaMappingController controller
E0620 08:05:00.830459    6766 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *api.PolicyBinding: Get https://AAA.BBB.CCC.DDD:8443/apis/authorization.openshift.io/v1/policybindings?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
E0620 08:05:00.835261    6766 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *api.Policy: Get https://AAA.BBB.CCC.DDD:8443/apis/authorization.openshift.io/v1/policies?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
E0620 08:05:00.835539    6766 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *api.ClusterPolicyBinding: Get https://AAA.BBB.CCC.DDD:8443/apis/authorization.openshift.io/v1/clusterpolicybindings?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
E0620 08:05:00.836570    6766 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *api.ClusterPolicy: Get https://AAA.BBB.CCC.DDD:8443/apis/authorization.openshift.io/v1/clusterpolicies?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
E0620 08:05:00.837038    6766 reflector.go:201] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45: Failed to list *api.ClusterResourceQuota: Get https://AAA.BBB.CCC.DDD:8443/apis/quota.openshift.io/v1/clusterresourcequotas?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
W0620 08:05:01.043705    6766 genericapiserver.go:295] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
I0620 08:05:01.314079    6766 openshift-apiserver.go:227] Starting Origin API at /apis/project.openshift.io/v1
I0620 08:05:01.317611    6766 openshift-apiserver.go:227] Starting Origin API at /apis/build.openshift.io/v1
I0620 08:05:01.324881    6766 openshift-apiserver.go:227] Starting Origin API at /apis/authorization.openshift.io/v1
I0620 08:05:01.328073    6766 openshift-apiserver.go:227] Starting Origin API at /apis/oauth.openshift.io/v1
I0620 08:05:01.329829    6766 openshift-apiserver.go:227] Starting Origin API at /apis/security.openshift.io/v1
I0620 08:05:01.558613    6766 openshift-apiserver.go:227] Starting Origin API at /apis/network.openshift.io/v1
I0620 08:05:01.561076    6766 openshift-apiserver.go:227] Starting Origin API at /apis/route.openshift.io/v1
I0620 08:05:01.563838    6766 openshift-apiserver.go:227] Starting Origin API at /apis/user.openshift.io/v1
I0620 08:05:01.567817    6766 openshift-apiserver.go:227] Starting Origin API at /apis/image.openshift.io/v1
I0620 08:05:01.570523    6766 openshift-apiserver.go:227] Starting Origin API at /apis/apps.openshift.io/v1
I0620 08:05:01.572260    6766 openshift-apiserver.go:227] Starting Origin API at /apis/template.openshift.io/v1
I0620 08:05:01.755206    6766 openshift-apiserver.go:227] Starting Origin API at /apis/quota.openshift.io/v1
I0620 08:05:01.928413    6766 openshift-apiserver.go:233] Started Origin API at /oapi/v1
E0620 08:05:01.970260    6766 reflector.go:201] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45: Failed to list *api.ClusterResourceQuota: Get https://AAA.BBB.CCC.DDD:8443/apis/quota.openshift.io/v1/clusterresourcequotas?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
E0620 08:05:01.991911    6766 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *api.ClusterPolicy: Get https://AAA.BBB.CCC.DDD:8443/apis/authorization.openshift.io/v1/clusterpolicies?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
E0620 08:05:01.992224    6766 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *api.ClusterPolicyBinding: Get https://AAA.BBB.CCC.DDD:8443/apis/authorization.openshift.io/v1/clusterpolicybindings?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
E0620 08:05:01.992345    6766 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *api.Policy: Get https://AAA.BBB.CCC.DDD:8443/apis/authorization.openshift.io/v1/policies?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
E0620 08:05:01.993627    6766 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *api.PolicyBinding: Get https://AAA.BBB.CCC.DDD:8443/apis/authorization.openshift.io/v1/policybindings?resourceVersion=0: dial tcp AAA.BBB.CCC.DDD:8443: getsockopt: connection refused
[restful] 2017/06/20 08:05:02 log.go:30: [restful/swagger] listing is available at https://AAA.BBB.CCC.DDD:8443/swaggerapi
[restful] 2017/06/20 08:05:02 log.go:30: [restful/swagger] https://AAA.BBB.CCC.DDD:8443/swaggerui/ is mapped to folder /swagger-ui/
I0620 08:05:02.483181    6766 serve.go:86] Serving securely on 0.0.0.0:8443
W0620 08:05:02.598837    6766 run_components.go:82] Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients
I0620 08:05:02.600351    6766 logs.go:41] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0]
I0620 08:05:02.600419    6766 logs.go:41] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0]
I0620 08:05:02.701612    6766 run_components.go:108] DNS listening at 0.0.0.0:8053
I0620 08:05:02.737281    6766 ensure.go:223] No cluster policy found.  Creating bootstrap policy based on: openshift.local.config/master/policy.json
I0620 08:05:03.356618    6766 trace.go:61] Trace "Create /api/v1/namespaces/default/services" (started 2017-06-20 08:05:02.529547434 +0000 UTC):
[18.731µs] [18.731µs] About to convert to expected version
[74.247µs] [55.516µs] Conversion done
[668.475248ms] [668.401001ms] About to store object in database
[826.958626ms] [158.483378ms] Object stored in database
[826.960666ms] [2.04µs] Self-link added
"Create /api/v1/namespaces/default/services" [826.994112ms] [33.446µs] END
W0620 08:05:03.377338    6766 lease_endpoint_reconciler.go:176] Resetting endpoints for master service "kubernetes" to [AAA.BBB.CCC.DDD]
I0620 08:05:13.208523    6766 ensure.go:208] Created default security context constraint privileged
I0620 08:05:13.213550    6766 ensure.go:208] Created default security context constraint nonroot
I0620 08:05:13.218318    6766 ensure.go:208] Created default security context constraint hostmount-anyuid
I0620 08:05:13.221744    6766 ensure.go:208] Created default security context constraint hostaccess
I0620 08:05:13.226588    6766 ensure.go:208] Created default security context constraint restricted
I0620 08:05:13.230435    6766 ensure.go:208] Created default security context constraint anyuid
I0620 08:05:13.235607    6766 ensure.go:208] Created default security context constraint hostnetwork
I0620 08:05:14.599503    6766 start_master.go:643] Controllers starting (*)
I0620 08:05:15.432208    6766 docker.go:364] Connecting to docker on unix:///var/run/docker.sock
I0620 08:05:15.433072    6766 docker.go:384] Start docker client with request timeout=2m0s
W0620 08:05:15.436757    6766 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
I0620 08:05:15.525973    6766 start_master.go:728] Started serviceaccount-token controller
I0620 08:05:15.586129    6766 nodecontroller.go:219] Sending events to api server.
I0620 08:05:15.587324    6766 taint_controller.go:157] Sending events to api server.
I0620 08:05:15.589358    6766 node_config.go:370] DNS Bind to AAA.BBB.CCC.DDD:53
I0620 08:05:15.589428    6766 start_node.go:345] Starting node subdomain.example.com (v3.6.0-alpha.2+2d90d52-359)
I0620 08:05:15.592536    6766 start_node.go:354] Connecting to API server https://AAA.BBB.CCC.DDD:8443
W0620 08:05:15.602947    6766 node.go:207] Error running 'chcon' to set the kubelet volume root directory SELinux context: exit status 1
I0620 08:05:15.603882    6766 docker.go:364] Connecting to docker on unix:///var/run/docker.sock
I0620 08:05:15.603959    6766 docker.go:384] Start docker client with request timeout=2m0s
I0620 08:05:15.608202    6766 node.go:143] Connecting to Docker at unix:///var/run/docker.sock
I0620 08:05:15.655371    6766 feature_gate.go:144] feature gates: map[]
I0620 08:05:15.656607    6766 manager.go:143] cAdvisor running in container: "/docker/75fec75eb7a94c61a55caef95332c5609d2cbfa4e427c46fc2a4688a0851cd4b"
I0620 08:05:15.681562    6766 node.go:364] Using iptables Proxier.
W0620 08:05:15.719020    6766 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
W0620 08:05:15.724558    6766 node.go:501] Failed to retrieve node info: nodes "subdomain.example.com" not found
W0620 08:05:15.724655    6766 proxier.go:309] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
W0620 08:05:15.724663    6766 proxier.go:314] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0620 08:05:15.724688    6766 node.go:393] Tearing down userspace rules.
I0620 08:05:16.018624    6766 fs.go:117] Filesystem partitions: map[/dev/vda1:{mountpoint:/var/lib/docker/aufs major:254 minor:1 fsType:ext3 blockSize:0}]
I0620 08:05:16.021960    6766 manager.go:198] Machine: {NumCores:1 CpuFrequency:2394452 MemoryCapacity:2057629696 MachineID:cdb7f92199c748a7a20d33ab52974cbb SystemUUID:EBF7B3D8-DAB9-47C4-BA0F-284565884230 BootID:447508a8-63c1-42bb-bd41-939ea709f1d3 Filesystems:[{Device:/dev/vda1 DeviceMajor:254 DeviceMinor:1 Capacity:10534313984 Type:vfs Inodes:655360 HasInodes:true} {Device:none DeviceMajor:0 DeviceMinor:43 Capacity:10534313984 Type:vfs Inodes:655360 HasInodes:true}] DiskMap:map[254:0:{Name:vda Major:254 Minor:0 Size:10737418240 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:fa:16:3e:f0:e0:fe Speed:0 Mtu:1500} {Name:tun0 MacAddress: Speed:10 Mtu:1500}] Topology:[{Id:0 Memory:2057629696 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I0620 08:05:16.152503    6766 manager.go:204] Version: {KernelVersion:3.16.0-4-amd64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:17.03.1-ce DockerAPIVersion:1.27 CadvisorVersion: CadvisorRevision:}
I0620 08:05:16.154539    6766 server.go:509] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
I0620 08:05:16.165932    6766 container_manager_linux.go:244] container manager verified user specified cgroup-root exists: /
I0620 08:05:16.166037    6766 container_manager_linux.go:249] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd ProtectKernelDefaults:false EnableCRI:true NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[]}
I0620 08:05:16.167827    6766 kubelet.go:265] Watching apiserver
W0620 08:05:16.281243    6766 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I0620 08:05:16.281363    6766 kubelet.go:494] Hairpin mode set to "hairpin-veth"
W0620 08:05:16.292849    6766 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
I0620 08:05:16.343728    6766 node.go:493] Started Kubernetes Proxy on 0.0.0.0
I0620 08:05:16.346155    6766 node.go:332] Starting DNS on AAA.BBB.CCC.DDD:53
I0620 08:05:16.346912    6766 logs.go:41] skydns: ready for queries on cluster.local. for tcp://AAA.BBB.CCC.DDD:53 [rcache 0]
I0620 08:05:16.346975    6766 logs.go:41] skydns: ready for queries on cluster.local. for udp://AAA.BBB.CCC.DDD:53 [rcache 0]
I0620 08:05:16.509299    6766 docker_service.go:184] Docker cri networking managed by kubernetes.io/no-op
F0620 08:05:16.627956    6766 node.go:297] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

The text was updated successfully, but these errors were encountered:

janmg · 2017-06-28T12:50:54Z

Kubernetes overrides of the default cfgroups and set to systemd instead. On systems where systemd is not used, this is unfortunate.

I compiled openshift origin again after commenting out line 79 in pkg/cmd/server/kubernetes/node/options/options.go
// server.CgroupDriver = "systemd"

... and now for me docker and kubernetes are using cgroupfs again for me.

Kubernetes does have a method to figure the docker cgroup driver in local-up-cluster.sh. But I don't know how everything is supposed to work together.
CGROUP_DRIVER=$(docker info | grep "Cgroup Driver:" | cut -f3- -d' ')

mcastelino · 2017-08-04T00:06:44Z

I get the same error even on Fedora

Fedora release 24 (Twenty Four)
4.13.0-0.rc2.git0.1.fc27.x86_64

docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:08:20 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:08:20 2017
 OS/Arch:      linux/amd64
 Experimental: false

node.go:282] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

janmg · 2017-08-04T12:59:36Z

Fedora is a frontrunner for SystemD, so if you have installed Docker using any of these methods, you should have a docker with a systemd cgroup driver. run "docker info" and your docker will show cgroupfs as cgroup driver. Probably you are better off reinstalling your docker using these instructions.
https://docs.docker.com/engine/installation/linux/docker-ce/fedora/

Debian is more recently moving towards systemd, but it's optional.

Alpine which I am running my openshift on, doesn't come with systemd. I have to patch the node_config.

mcastelino · 2017-08-04T16:21:30Z

@janmg you are right. I had overridden the default system unit file to add a second runtime. And that causes dockerd to switch back to cgroup. Adding systemd explicitly fixes the issue.

The unit file in my case, that works

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd  -D  --add-runtime cor=/bin/cc-oci-runtime --default-runtime=runc --exec-opt native.cgroupdriver=systemd

[Service]
# Allow maximum number of containers to run.
TasksMax=infinity

EamonZhang · 2017-12-13T02:34:00Z

openshift version
openshift v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

F1213 10:15:19.522499   21099 node.go:264] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

the default is not to use systemd for cgroups because the delegate issues still
 exists and systemd currently does not support the cgroup feature set required
 for containers run by docker

chenray844 · 2018-01-09T11:23:15Z

F0109 06:13:41.401134   11461 node.go:264] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

./openshift version
openshift v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

docker version
Client:
 Version:	17.12.0-ce
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	c97c6d6
 Built:	Wed Dec 27 20:10:14 2017
 OS/Arch:	linux/amd64

Server:
 Engine:
  Version:	17.12.0-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.2
  Git commit:	c97c6d6
  Built:	Wed Dec 27 20:12:46 2017
  OS/Arch:	linux/amd64
  Experimental:	false

lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.4.1708 (Core) 
Release:	7.4.1708
Codename:	Core

can some one give me some advice to fix F0109 06:13:41.401134 11461 node.go:264] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs" error?

nazarewk · 2018-01-20T19:31:38Z

Same here.

openshift v3.7.1+ab0f056
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

Tried on following systems (uname -a):

Arch Linux Linux krna 4.14.13-1-ARCH #1 SMP PREEMPT Wed Jan 10 11:14:50 UTC 2018 x86_64 GNU/Linux
CoreOS Linux localhost 4.14.11-coreos #1 SMP Fri Jan 5 11:00:14 UTC 2018 x86_64 Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz GenuineIntel GNU/Linux

greut · 2018-01-21T08:33:18Z

@nazarewk @chenray844 follow @mcastelino advice --exec-opt native.cgroupdriver=systemd

davidblum · 2018-01-31T02:15:07Z

Ran into this exact issue tonight --exec-opt native.cgroupdriver=systemd in /usr/lib/systemd/system/docker.service with: ExecStart=/usr/bin/dockerd --exec-opt native.cgroupdriver=systemd+ docker and kubelet restarts, got this working for me.

Running:

[hostname]# uname -a
Linux hostname.net 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[hostname]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

openshift-bot · 2018-05-01T05:53:27Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2018-05-31T06:40:04Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2018-06-30T06:40:56Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

mfojtik assigned derekwaynecarr Jun 20, 2017

mfojtik added component/kubernetes kind/bug Categorizes issue or PR as related to a bug. priority/P2 labels Jun 20, 2017

queer mentioned this issue Jul 25, 2017

Creating any pod fails to mount volumes: MountVolume.SetUp failed for volume "deployer-token-50rds" : exit status 32 #15441

Closed

queer mentioned this issue Jan 22, 2018

missing service/route (service "webconsole" not found) #18207

Closed

jmwilkinson mentioned this issue Mar 9, 2018

How to override default "cgroup-driver" setting #18928

Closed

openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 1, 2018

openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 31, 2018

openshift-ci-robot closed this as completed Jun 30, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

openshift/origin docker container fails to start: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs" #14766

openshift/origin docker container fails to start: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs" #14766

queer commented Jun 20, 2017 •

edited

janmg commented Jun 28, 2017 •

edited

mcastelino commented Aug 4, 2017

janmg commented Aug 4, 2017

mcastelino commented Aug 4, 2017

EamonZhang commented Dec 13, 2017

chenray844 commented Jan 9, 2018

nazarewk commented Jan 20, 2018

greut commented Jan 21, 2018

davidblum commented Jan 31, 2018

openshift-bot commented May 1, 2018

openshift-bot commented May 31, 2018

openshift-bot commented Jun 30, 2018

openshift/origin docker container fails to start: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs" #14766

openshift/origin docker container fails to start: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs" #14766

Comments

queer commented Jun 20, 2017 • edited

Version

Steps To Reproduce

Current Result

Expected Result

Additional Information

janmg commented Jun 28, 2017 • edited

mcastelino commented Aug 4, 2017

janmg commented Aug 4, 2017

mcastelino commented Aug 4, 2017

EamonZhang commented Dec 13, 2017

chenray844 commented Jan 9, 2018

nazarewk commented Jan 20, 2018

greut commented Jan 21, 2018

davidblum commented Jan 31, 2018

openshift-bot commented May 1, 2018

openshift-bot commented May 31, 2018

openshift-bot commented Jun 30, 2018

queer commented Jun 20, 2017 •

edited

janmg commented Jun 28, 2017 •

edited