systemd-udevd takes high CPU on CoreOS stable v835.9.0 #1036
Comments
can you provide a full log of systemd-udevd?
|
Here you go:
|
@ntquyen what platform are you on? if possible, can you reproduce this in CoreOS alpha (891.0.0)? |
I'm using CoreOS inside our in-house data center. I had this issue before with alpha versions prior to 891.0.0 (3 months ago). This issue is gone when I switched to v773.1.0 stable. But then stable version upgraded to v835.9.0 and I got this issue again |
@ntquyen When you're watching top on one of these malfunctioning systems, do you observe a constant churn of systemd-udevd processes? It should be visible by watching the PID values of the systemd-udevd processes. If they're continuously increasing then the answer is yes. I'd just like to know if there's a finite set of long-running systemd-udevd processes burning cpu, or if there's a continuous stream of short-lived systemd-udevd processes being created by the main systemd-udevd process. systemd-udevd has a multiprocess execution model, but the new processes are forked in response to events, so this may be just a consequence of systemd-udevd getting flooded with events. The command Additionally, running systemd-udevd with --debug may help shed some light on what's going on, if |
Thanks @vcaputo for you debug instructions! I will check at let you know. |
The PIDs of systemd-udevd processes are constant, they are not increasing.
The command
I tried
|
Running strace on one of the
|
@ntquyen This doesn't appear to be a systemd-udevd bug, it looks like this virtual device is legitimately (from the systemd-udevd perspective) constantly being plugged/unplugged from the kernel, and systemd-udevd is simply responding to that. Good news is you're not alone: Bad news is it doesn't appear to have been solved yet. |
Thanks @vcaputo for the shared links! I'm running CoreOS in Hyper-V as well.. May be I'll rollback to v773.1.0 and wait for the bug fixed. |
@ntquyen can you try the latest Alpha? We just bumped to the 4.4 kernel. |
Thanks @crawford, I will try and let you know! |
@crawford the latest alpha still has this issue. |
OK. Thanks for testing. The Hyper-V guys are looking into this, so we'll see what they have to say. |
This kernel bug has been fixed. Can't wait to see it in next alpha |
torvalds/linux@ed9ba60 is currently only in linux v4.5. it is possible to backport it, though. |
A much more simple solution: disabling Volume Snapshot for the VM in Hyper-V. |
Should be fixed by coreos/coreos-overlay#1867. |
After upgraded to v835.9.0 stable there are a lot of
systemd-udevd
taking very high CPU that make the OS slow performingSystem log error related to
systemd-udevd
:I'm using CoreOS stable v835.9.0 with kubelet running v1.0.1
The text was updated successfully, but these errors were encountered: