lxcfs.service exited with code=killed, status=11/SEGV

[pipsqueakup]

My lxcfs is managed by systemd. However every time when system reboot, i encountered this error just about one minute after lxcfs started

Nov 22 17:08:46 izbp1a20q8kyn1fvlbkfz7z kernel: lxcfs3.0.1[671]: segfault at f1c ip 00007fe6dec7f4b0 sp 00007fe6de03b6a0 error 4 in ld-2.17.so[7fe6dec76000+21000]
Nov 22 17:08:46 izbp1a20q8kyn1fvlbkfz7z systemd: lxcfs.service: main process exited, code=killed, status=11/SEGV
Nov 22 17:08:46 izbp1a20q8kyn1fvlbkfz7z systemd: Unit lxcfs.service entered failed state.
Nov 22 17:08:46 izbp1a20q8kyn1fvlbkfz7z systemd: lxcfs.service failed.
Nov 22 17:08:47 izbp1a20q8kyn1fvlbkfz7z systemd: lxcfs.service holdoff time over, scheduling restart.
Nov 22 17:08:47 izbp1a20q8kyn1fvlbkfz7z systemd: Starting FUSE filesystem for LXC...

after scheduled restart, it's ok.

My lxcfs.service is as follows

[Unit]
Description=FUSE filesystem for LXC
ConditionVirtualization=!container
Documentation=man:lxcfs(1)
Before=kubelet.service

[Service]
ExecStart=/usr/local/bin/start-lxcfs.sh
KillMode=process
Restart=always
ExecStopPost=-/usr/bin/fusermount -u /var/lib/lxc/lxcfs
Delegate=yes

ExecStartPost=/usr/local/bin/remount-lxcfs.sh

[Install]
WantedBy=multi-user.target

Has anyone has the same problem?

[tomponline]

@pipsqueakup what OS/kernel do you use?

[pipsqueakup]

@pipsqueakup what OS/kernel do you use?

LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.4.1708 (Core)
Release: 7.4.1708
Codename: Core

Linux version 3.10.0-693.2.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Sep 12 22:26:13 UTC 2017

[pipsqueakup]

After some tests, i figured out lxcfs crashes when the first pod is schedule to the host machine. To be more specific, the host machine runs dockerd, kubelet and lxcfs on it, every time the machine reboots, lxcfs, docker and kubelet all started successfully, when a pod is scheduled to it, lxcfs crashes immediately, but after restarts by systemd ,it worked fine.

My lxcfs version is 3.0.1, Kubernetes v1.9.3, Docker version 17.06.2-ce, build cec0b72

Below is the specific log

Nov 22 19:10:29 izbp156kyf4dxq6gux81foz kubelet: I1122 19:10:29.762958    1456 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "lxcfs-proc-cpuinfo" (UniqueName: "kubernetes.io/host-path/38ef652d-ee47-11e8-8d73-00163e0d9de0-lxcfs-proc-cpuinfo") pod "test7-0" (UID: "38ef652d-ee47-11e8-8d73-00163e0d9de0")
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz kubelet: I1122 19:10:29.763001    1456 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "resolv-conf" (UniqueName: "kubernetes.io/configmap/38ef652d-ee47-11e8-8d73-00163e0d9de0-resolv-conf") pod "test7-0" (UID: "38ef652d-ee47-11e8-8d73-00163e0d9de0")
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz kubelet: I1122 19:10:29.763018    1456 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "lxcfs-proc-meminfo" (UniqueName: "kubernetes.io/host-path/38ef652d-ee47-11e8-8d73-00163e0d9de0-lxcfs-proc-meminfo") pod "test7-0" (UID: "38ef652d-ee47-11e8-8d73-00163e0d9de0")
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz kubelet: I1122 19:10:29.763036    1456 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "lxcfs-proc-swaps" (UniqueName: "kubernetes.io/host-path/38ef652d-ee47-11e8-8d73-00163e0d9de0-lxcfs-proc-swaps") pod "test7-0" (UID: "38ef652d-ee47-11e8-8d73-00163e0d9de0")
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz kubelet: I1122 19:10:29.763054    1456 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "lxcfs-proc-uptime" (UniqueName: "kubernetes.io/host-path/38ef652d-ee47-11e8-8d73-00163e0d9de0-lxcfs-proc-uptime") pod "test7-0" (UID: "38ef652d-ee47-11e8-8d73-00163e0d9de0")
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz kubelet: I1122 19:10:29.763074    1456 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-w558x" (UniqueName: "kubernetes.io/secret/38ef652d-ee47-11e8-8d73-00163e0d9de0-default-token-w558x") pod "test7-0" (UID: "38ef652d-ee47-11e8-8d73-00163e0d9de0")
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz kubelet: I1122 19:10:29.763092    1456 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "lxcfs-proc-stat" (UniqueName: "kubernetes.io/host-path/38ef652d-ee47-11e8-8d73-00163e0d9de0-lxcfs-proc-stat") pod "test7-0" (UID: "38ef652d-ee47-11e8-8d73-00163e0d9de0")
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz kubelet: I1122 19:10:29.763110    1456 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "lxcfs-proc-diskstats" (UniqueName: "kubernetes.io/host-path/38ef652d-ee47-11e8-8d73-00163e0d9de0-lxcfs-proc-diskstats") pod "test7-0" (UID: "38ef652d-ee47-11e8-8d73-00163e0d9de0")
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz kubelet: I1122 19:10:29.763130    1456 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "lxcfs-lxc-shared" (UniqueName: "kubernetes.io/host-path/38ef652d-ee47-11e8-8d73-00163e0d9de0-lxcfs-lxc-shared") pod "test7-0" (UID: "38ef652d-ee47-11e8-8d73-00163e0d9de0")
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz systemd: Started Kubernetes transient mount for /var/lib/kubelet/pods/38ef652d-ee47-11e8-8d73-00163e0d9de0/volumes/kubernetes.io~secret/default-token-w558x.
Nov 22 19:10:29 izbp156kyf4dxq6gux81foz systemd: Starting Kubernetes transient mount for /var/lib/kubelet/pods/38ef652d-ee47-11e8-8d73-00163e0d9de0/volumes/kubernetes.io~secret/default-token-w558x.
Nov 22 19:10:30 izbp156kyf4dxq6gux81foz kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Nov 22 19:10:30 izbp156kyf4dxq6gux81foz kernel: cni0: port 4(veth6df1ffda) entered blocking state
Nov 22 19:10:30 izbp156kyf4dxq6gux81foz kernel: cni0: port 4(veth6df1ffda) entered disabled state
Nov 22 19:10:30 izbp156kyf4dxq6gux81foz kernel: device veth6df1ffda entered promiscuous mode
Nov 22 19:10:30 izbp156kyf4dxq6gux81foz kernel: cni0: port 4(veth6df1ffda) entered blocking state
Nov 22 19:10:30 izbp156kyf4dxq6gux81foz kernel: cni0: port 4(veth6df1ffda) entered forwarding state
Nov 22 19:10:30 izbp156kyf4dxq6gux81foz kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Nov 22 19:10:50 izbp156kyf4dxq6gux81foz kernel: lxcfs3.0.1[520]: segfault at f1c ip 00007f4d3cf214b0 sp 00007f4d3badc6a0 error 4 in ld-2.17.so[7f4d3cf18000+21000]
Nov 22 19:10:50 izbp156kyf4dxq6gux81foz systemd: lxcfs.service: main process exited, code=killed, status=11/SEGV
Nov 22 19:10:50 izbp156kyf4dxq6gux81foz systemd: Unit lxcfs.service entered failed state.
Nov 22 19:10:50 izbp156kyf4dxq6gux81foz systemd: lxcfs.service failed.
Nov 22 19:10:50 izbp156kyf4dxq6gux81foz dockerd: time="2018-11-22T19:10:50.208405892+08:00" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:262: starting container process caused \"process_linux.go:339: container init caused \\\"rootfs_linux.go:57: mounting \\\\\\\"/var/lib/lxc/lxcfs/proc/swaps\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/55aeb2b28e30c9eb2c2010e51e7144fd20bd3f4202b1e3b936cedc70a8b02b1f/merged\\\\\\\" at \\\\\\\"/proc/swaps\\\\\\\" caused \\\\\\\"stat /var/lib/lxc/lxcfs/proc/swaps: no such file or directory\\\\\\\"\\\"\"\n" id=4af21d52400517688e0131bb6bc9846d74d64e7ed5147e3a626c53c7ff277965

[tomponline]

@pipsqueakup its been my experience that lxcfs is not stable on CentOS 7 with the vanilla kernel.

I would first try installing the elrepo mainline kernel, or you could try my "latest long term" kernel, based on the same package.

See #252 and https://www.tomp.uk/blog/view/40

[pipsqueakup]

@pipsqueakup its been my experience that lxcfs is not stable on CentOS 7 with the vanilla kernel.

I would first try installing the elrepo mainline kernel, or you could try my "latest long term" kernel, based on the same package.

See #252 and https://www.tomp.uk/blog/view/40

@tomponline i found the reason, two lxcfs process are started and they use the same library /usr/local/lib/lxcfs/liblxcfs.so. The latterly started process would affect the formerly started.

[tomponline]

good news @pipsqueakup

I didnt know that docker could work with lxcfs? Are you running lxc also?

[hallyn]

Why are two lxcfs processess started? Is it a bug in lxcfs itself? Or should this issue be closed?

[stgraber]

Doesn't sound like a bug in lxcfs proper. If two processes are spawned by Docker or something else and use the same paths, then bad things like this will happen.