New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-udevd takes high CPU on CoreOS stable v835.9.0 #1036

Closed
ntquyen opened this Issue Dec 15, 2015 · 18 comments

Comments

Projects
None yet
4 participants
@ntquyen

ntquyen commented Dec 15, 2015

After upgraded to v835.9.0 stable there are a lot of systemd-udevd taking very high CPU that make the OS slow performing

screen shot 2015-12-15 at 14 10 04

System log error related to systemd-udevd:

$ dmesg
...
systemd-udevd[57033]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
...

I'm using CoreOS stable v835.9.0 with kubelet running v1.0.1

@mischief

This comment has been minimized.

mischief commented Dec 15, 2015

can you provide a full log of systemd-udevd?

journalctl -b -u systemd-udevd.service

@ntquyen

This comment has been minimized.

ntquyen commented Dec 16, 2015

Here you go:

$ sudo journalctl -b -u systemd-udevd.service

-- Logs begin at Sun 2015-12-13 05:13:50 UTC, end at Wed 2015-12-16 02:30:35 UTC. --
Dec 15 02:44:40 localhost systemd[1]: Starting udev Kernel Device Manager...
Dec 15 02:44:40 localhost systemd-udevd[228]: RUN{builtin}: 'uaccess' unknown /usr/lib64/udev/rules.d/73-seat-late.rules:15
Dec 15 02:44:40 localhost systemd[1]: Started udev Kernel Device Manager.
Dec 15 02:44:41 localhost systemd-udevd[228]: RUN{builtin}: 'uaccess' unknown /usr/lib64/udev/rules.d/73-seat-late.rules:15
Dec 15 02:46:37 localhost systemd-udevd[275]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:46:40 localhost systemd-udevd[285]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:46:40 localhost systemd-udevd[285]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:46:40 localhost systemd-udevd[285]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:46:42 localhost systemd-udevd[295]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:46:42 localhost systemd-udevd[295]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:46:47 localhost systemd-udevd[1405]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:46:47 localhost systemd-udevd[1405]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:46:48 localhost systemd[1]: Stopping udev Kernel Device Manager...
Dec 15 02:46:48 localhost systemd[1]: Stopped udev Kernel Device Manager.
Dec 15 02:47:01 dev-coreos-k8s_01 systemd-udevd[1713]: RUN{builtin}: 'uaccess' unknown /usr/lib64/udev/rules.d/73-seat-late.rules:15
Dec 15 02:47:02 dev-coreos-k8s_01 systemd-udevd[1713]: RUN{builtin}: 'uaccess' unknown /usr/lib64/udev/rules.d/73-seat-late.rules:15
Dec 15 02:47:11 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:11 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:16 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:16 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:16 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:16 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:16 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:17 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:17 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:18 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:18 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:23 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:23 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:32 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:32 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:32 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:34 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:34 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:36 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:41 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:41 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
Dec 15 02:47:43 dev-coreos-k8s_01 systemd-udevd[1743]: setting mode of /dev/vmbus/hv_vss to 020600 failed: No such file or directory
Dec 15 02:47:43 dev-coreos-k8s_01 systemd-udevd[1743]: setting owner of /dev/vmbus/hv_vss to uid=0, gid=0 failed: No such file or directory
...
@mischief

This comment has been minimized.

mischief commented Dec 16, 2015

@ntquyen what platform are you on? if possible, can you reproduce this in CoreOS alpha (891.0.0)?

@ntquyen

This comment has been minimized.

ntquyen commented Dec 16, 2015

I'm using CoreOS inside our in-house data center. I had this issue before with alpha versions prior to 891.0.0 (3 months ago). This issue is gone when I switched to v773.1.0 stable. But then stable version upgraded to v835.9.0 and I got this issue again

@vcaputo

This comment has been minimized.

vcaputo commented Dec 16, 2015

@ntquyen When you're watching top on one of these malfunctioning systems, do you observe a constant churn of systemd-udevd processes? It should be visible by watching the PID values of the systemd-udevd processes. If they're continuously increasing then the answer is yes.

I'd just like to know if there's a finite set of long-running systemd-udevd processes burning cpu, or if there's a continuous stream of short-lived systemd-udevd processes being created by the main systemd-udevd process.

systemd-udevd has a multiprocess execution model, but the new processes are forked in response to events, so this may be just a consequence of systemd-udevd getting flooded with events. The command udevadm monitor may also help shed some light on this.

Additionally, running systemd-udevd with --debug may help shed some light on what's going on, if udevadm monitor isn't helpful.

@mischief mischief added this to the CoreOS 835.10.0 milestone Dec 17, 2015

@ntquyen

This comment has been minimized.

ntquyen commented Dec 18, 2015

Thanks @vcaputo for you debug instructions! I will check at let you know.

@ntquyen

This comment has been minimized.

ntquyen commented Dec 21, 2015

The PIDs of systemd-udevd processes are constant, they are not increasing.

core@dev-coreos-k8s_02 ~ $ ps aux | grep systemd-udevd
root      1686 72.9  1.0  73712 39740 ?        Rs   Dec17 4399:57 /usr/lib/systemd/systemd-udevd
root      1897 10.6  0.3  59648 12864 ?        R    Dec17 643:16 /usr/lib/systemd/systemd-udevd
root      1898  0.0  0.0  59648  2188 ?        S    Dec17   0:00 /usr/lib/systemd/systemd-udevd
root      1899  0.0  0.0  59648  2556 ?        S    Dec17   0:00 /usr/lib/systemd/systemd-udevd
root      1900  0.3  0.5  59648 23492 ?        S    Dec17  24:03 /usr/lib/systemd/systemd-udevd
root      1901  0.0  0.5  59648 23364 ?        S    Dec17   0:15 /usr/lib/systemd/systemd-udevd
root      1902  0.0  0.0  59648  3120 ?        S    Dec17   0:01 /usr/lib/systemd/systemd-udevd
root      1906  7.1  0.5  59648 23492 ?        S    Dec17 428:20 /usr/lib/systemd/systemd-udevd
root      1907  0.0  0.5  59648 23492 ?        S    Dec17   5:09 /usr/lib/systemd/systemd-udevd
root      1908  0.0  0.3  59648 13276 ?        S    Dec17   0:00 /usr/lib/systemd/systemd-udevd
root      1909  0.0  0.0  59648  2164 ?        S    Dec17   0:00 /usr/lib/systemd/systemd-udevd
root      1910  0.0  0.5  59780 23368 ?        S    Dec17   1:07 /usr/lib/systemd/systemd-udevd
root      1911  0.0  0.5  59780 23376 ?        S    Dec17   0:00 /usr/lib/systemd/systemd-udevd
root      2954  1.4  0.0  66344  2632 ?        S    Dec17  83:03 /usr/lib/systemd/systemd-udevd
root     17692  4.0  0.5  69412 22884 ?        S    Dec17 241:51 /usr/lib/systemd/systemd-udevd
root     60802  0.0  0.0  63160  2164 ?        S    Dec17   0:00 /usr/lib/systemd/systemd-udevd
root     60804  0.0  0.0  60784  2568 ?        S    Dec17   0:02 /usr/lib/systemd/systemd-udevd

The command udevadm monitor print repeatedly a series of log:

core@dev-coreos-k8s_02 ~ $ systemctl status systemd-udevd
...
KERNEL[362225.203383] remove   /devices/virtual/misc/vmbus!hv_vss (misc)
UDEV  [362225.203424] add      /devices/virtual/misc/vmbus!hv_vss (misc)
KERNEL[362225.203511] remove   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/vmbus_102964226 (vmbus)
KERNEL[362225.203929] add      /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/vmbus_102964227 (vmbus)
UDEV  [362225.204314] remove   /devices/virtual/misc/vmbus!hv_vss (misc)
UDEV  [362225.206125] add      /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/vmbus_102850814 (vmbus)
KERNEL[362225.206678] add      /devices/virtual/misc/vmbus!hv_vss (misc)
UDEV  [362225.207461] add      /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/vmbus_102850815 (vmbus)
KERNEL[362225.207603] remove   /devices/virtual/misc/vmbus!hv_vss (misc)
KERNEL[362225.207648] remove   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/vmbus_102964227 (vmbus)
KERNEL[362225.208236] add      /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/vmbus_102964228 (vmbus)
KERNEL[362225.208397] add      /devices/virtual/misc/vmbus!hv_vss (misc)
UDEV  [362225.209317] remove   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/vmbus_102850814 (vmbus)
UDEV  [362225.209941] remove   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/vmbus_102850815 (vmbus)
KERNEL[362225.210097] remove   /devices/virtual/misc/vmbus!hv_vss (misc)
KERNEL[362225.210164] remove   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/vmbus_102964228 (vmbus)
UDEV  [362225.210431] add      /devices/virtual/misc/vmbus!hv_vss (misc)
UDEV  [362225.210987] remove   /devices/virtual/misc/vmbus!hv_vss (misc)
...

I tried udevadm monitor in v773.1.0 which doesn't have this issue:

core@dev-coreos-k8s_03 ~ $ udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent


@ntquyen

This comment has been minimized.

ntquyen commented Dec 21, 2015

Running strace on one of the systemd-udevd processes, looks like it is looping:

core@dev-coreos-k8s_02 ~ $ sudo strace -fvvp 1686
Process 1686 attached
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=-1279045853, groups=00000000}, msg_iov(2)=[{"libudev\0\376\355\312\376(\0\0\0(\0\0\0\321\0\0\0\0308\374\315\0\0\0\0"..., 40}, {"ACTION=add\0DEVPATH=/devices/LNXS"..., 209}], msg_controllen=0, msg_flags=0}, 0) = 249
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 17, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685795, 980578429}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 579203027}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 579251726}) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=1906, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=1897, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, 0x7fff0d47a290, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=1686, groups=00000000}, msg_iov(2)=[{"libudev\0\376\355\312\376(\0\0\0(\0\0\0\243\0\0\0K\32Nu\0\0\0\0"..., 40}, {"ACTION=remove\0DEVPATH=/devices/v"..., 163}], msg_controllen=0, msg_flags=0}, 0) = 203
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 15, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685795, 985843143}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 584346542}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 584425141}) = 0
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 15, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685795, 986063240}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 584556239}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 584601638}) = 0
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000001}, msg_iov(1)=[{"add@/devices/virtual/misc/vmbus!"..., 8192}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=0, uid=0, gid=0}}, msg_flags=0}, 0) = 164
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af0727a3e0, 0x5, 0x7f534078e858) = 0x10
clock_gettime(CLOCK_MONOTONIC, {366112, 584828734}) = 0
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af0692bbe0, 0x5, 0x7f534078e820) = 0x10
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af06461290, 0x5, 0x7f534078e7e8) = 0x10
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 15, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685795, 993953210}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 592917602}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 593379294}) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=1897, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=2954, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=17692, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, 0x7fff0d47a290, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=1686, groups=00000000}, msg_iov(2)=[{"libudev\0\376\355\312\376(\0\0\0(\0\0\0\240\0\0\0K\32Nu\0\0\0\0"..., 40}, {"ACTION=add\0DEVPATH=/devices/virt"..., 160}], msg_controllen=0, msg_flags=0}, 0) = 200
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=-1333008039, groups=00000000}, msg_iov(2)=[{"libudev\0\376\355\312\376(\0\0\0(\0\0\0\324\0\0\0\0308\374\315\0\0\0\0"..., 40}, {"ACTION=remove\0DEVPATH=/devices/L"..., 212}], msg_controllen=0, msg_flags=0}, 0) = 252
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 13, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685795, 998048843}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 596568042}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 596622541}) = 0
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 13, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685795, 998583535}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 597109833}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 597168032}) = 0
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000001}, msg_iov(1)=[{"remove@/devices/virtual/misc/vmb"..., 8192}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=0, uid=0, gid=0}}, msg_flags=0}, 0) = 170
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af06a85c60, 0x5, 0x55af05eaebe0) = 0x10
clock_gettime(CLOCK_MONOTONIC, {366112, 597701824}) = 0
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af063b3e50, 0x5, 0x55af05eaeba8) = 0x10
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af05ed5390, 0x5, 0x55af05eaeb70) = 0x10
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 13, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685796, 1850481}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 600380780}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 600419879}) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=1897, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=17692, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, 0x7fff0d47a290, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=1686, groups=00000000}, msg_iov(2)=[{"libudev\0\376\355\312\376(\0\0\0(\0\0\0\243\0\0\0K\32Nu\0\0\0\0"..., 40}, {"ACTION=remove\0DEVPATH=/devices/v"..., 163}], msg_controllen=0, msg_flags=0}, 0) = 203
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 11, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685796, 4212042}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 602761941}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 602814140}) = 0
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 11, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685796, 4459938}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 602971337}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 603040736}) = 0
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000001}, msg_iov(1)=[{"remove@/devices/LNXSYSTM:00/LNXS"..., 8192}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=0, uid=0, gid=0}}, msg_flags=0}, 0) = 268
syscall_318(0x55af059d1930, 0x10, 0x1, 0, 0x5, 0x7f5342a87438) = 0x10
clock_gettime(CLOCK_MONOTONIC, {366112, 603511129}) = 0
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af073e3600, 0x5, 0x7f5342a87400) = 0x10
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af0780e0d0, 0x5, 0x7f5342a873c8) = 0x10
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=-1333008039, groups=00000000}, msg_iov(2)=[{"libudev\0\376\355\312\376(\0\0\0(\0\0\0\324\0\0\0\0308\374\315\0\0\0\0"..., 40}, {"ACTION=remove\0DEVPATH=/devices/L"..., 212}], msg_controllen=0, msg_flags=0}, 0) = 252
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 13, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685796, 9337858}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 607847358}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 607915056}) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=1897, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=17692, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, 0x7fff0d47a290, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=1686, groups=00000000}, msg_iov(2)=[{"libudev\0\376\355\312\376(\0\0\0(\0\0\0\240\0\0\0K\32Nu\0\0\0\0"..., 40}, {"ACTION=add\0DEVPATH=/devices/virt"..., 160}], msg_controllen=0, msg_flags=0}, 0) = 200
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}}, 11, 0) = 1
clock_gettime(CLOCK_REALTIME, {1450685796, 11322826}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 609829425}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 609879724}) = 0
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}}, 11, 0) = 1
clock_gettime(CLOCK_REALTIME, {1450685796, 11536822}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 610074521}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 610125920}) = 0
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000001}, msg_iov(1)=[{"add@/devices/LNXSYSTM:00/LNXSYBU"..., 8192}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=0, uid=0, gid=0}}, msg_flags=0}, 0) = 262
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af05f9b5c0, 0x5, 0x7f5342a873c8) = 0x10
clock_gettime(CLOCK_MONOTONIC, {366112, 610453015}) = 0
syscall_318(0x55af059d1930, 0x10, 0x1, 0, 0x5, 0x7f5342a87400) = 0x10
syscall_318(0x55af059d1930, 0x10, 0x1, 0, 0x5, 0x7f5342a87438) = 0x10
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=-1333008039, groups=00000000}, msg_iov(2)=[{"libudev\0\376\355\312\376(\0\0\0(\0\0\0\321\0\0\0\0308\374\315\0\0\0\0"..., 40}, {"ACTION=add\0DEVPATH=/devices/LNXS"..., 209}], msg_controllen=0, msg_flags=0}, 0) = 249
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}}, 13, 0) = 1
clock_gettime(CLOCK_REALTIME, {1450685796, 15509457}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 614039756}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 614091255}) = 0
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}}, 13, 0) = 1
clock_gettime(CLOCK_REALTIME, {1450685796, 15771553}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 614282152}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 614335451}) = 0
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000001}, msg_iov(1)=[{"add@/devices/virtual/misc/vmbus!"..., 8192}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=0, uid=0, gid=0}}, msg_flags=0}, 0) = 164
syscall_318(0x55af059d1930, 0x10, 0x1, 0, 0x5, 0x55af05eba640) = 0x10
clock_gettime(CLOCK_MONOTONIC, {366112, 614576947}) = 0
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af05f2a3e0, 0x5, 0x55af05eba608) = 0x10
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af05fd0dc0, 0x5, 0x55af05eba5d0) = 0x10
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 13, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685796, 17155230}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 615732528}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 615800627}) = 0
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 13, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685796, 17477925}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 615991724}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 616140722}) = 0
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000001}, msg_iov(1)=[{"remove@/devices/virtual/misc/vmb"..., 8192}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=0, uid=0, gid=0}}, msg_flags=0}, 0) = 170
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af066d3350, 0x5, 0x7f53429cdee8) = 0x10
clock_gettime(CLOCK_MONOTONIC, {366112, 616437917}) = 0
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af05eceed0, 0x5, 0x7f53429cdf20) = 0x10
syscall_318(0x55af059d1930, 0x10, 0x1, 0x55af069ad9e0, 0x5, 0x7f53429cdf58) = 0x10
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 13, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685796, 20259080}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 618767579}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 619048474}) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=17692, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"", 0}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=1897, uid=0, gid=0}}, msg_flags=0}, MSG_DONTWAIT) = 0
recvmsg(7, 0x7fff0d47a290, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=1686, groups=00000000}, msg_iov(2)=[{"libudev\0\376\355\312\376(\0\0\0(\0\0\0\243\0\0\0K\32Nu\0\0\0\0"..., 40}, {"ACTION=remove\0DEVPATH=/devices/v"..., 163}], msg_controllen=0, msg_flags=0}, 0) = 203
epoll_wait(10, {{EPOLLIN, {u32=97704288, u64=94210205342048}}, {EPOLLIN, {u32=97704528, u64=94210205342288}}}, 11, 0) = 2
clock_gettime(CLOCK_REALTIME, {1450685796, 22410644}) = 0
clock_gettime(CLOCK_MONOTONIC, {366112, 621149140}) = 0
clock_gettime(CLOCK_BOOTTIME, {366112, 621210139}) = 0
...
...
^CProcess 1686 detached

@vcaputo

This comment has been minimized.

vcaputo commented Dec 21, 2015

@ntquyen This doesn't appear to be a systemd-udevd bug, it looks like this virtual device is legitimately (from the systemd-udevd perspective) constantly being plugged/unplugged from the kernel, and systemd-udevd is simply responding to that.

Good news is you're not alone:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1496927
http://www.linuxquestions.org/questions/fedora-35/high-cpu-load-from-systemd-udevd-under-hyper-v-after-kernel-update-4175561745/

Bad news is it doesn't appear to have been solved yet.

@ntquyen

This comment has been minimized.

ntquyen commented Dec 21, 2015

Thanks @vcaputo for the shared links! I'm running CoreOS in Hyper-V as well.. May be I'll rollback to v773.1.0 and wait for the bug fixed.

@crawford

This comment has been minimized.

Member

crawford commented Jan 13, 2016

@ntquyen can you try the latest Alpha? We just bumped to the 4.4 kernel.

@crawford crawford modified the milestones: CoreOS 877.2.0, CoreOS 877.3.0 Jan 13, 2016

@ntquyen

This comment has been minimized.

ntquyen commented Jan 16, 2016

Thanks @crawford, I will try and let you know!

@ntquyen

This comment has been minimized.

ntquyen commented Jan 19, 2016

@crawford the latest alpha still has this issue.

@crawford

This comment has been minimized.

Member

crawford commented Jan 19, 2016

OK. Thanks for testing. The Hyper-V guys are looking into this, so we'll see what they have to say.

@crawford crawford removed this from the CoreOS 877.3.0 milestone Jan 21, 2016

@ntquyen

This comment has been minimized.

ntquyen commented Mar 22, 2016

This kernel bug has been fixed. Can't wait to see it in next alpha 😄
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1496927

@mischief

This comment has been minimized.

mischief commented Mar 22, 2016

torvalds/linux@ed9ba60 is currently only in linux v4.5. it is possible to backport it, though.

@ntquyen

This comment has been minimized.

ntquyen commented Mar 29, 2016

A much more simple solution: disabling Volume Snapshot for the VM in Hyper-V.

@crawford

This comment has been minimized.

Member

crawford commented Apr 7, 2016

Should be fixed by coreos/coreos-overlay#1867.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment