sys-usb kworker events CPU consumption

[neowutran]

Qubes OS version
R4.0

Affected component(s) or functionality
USB device handeling

Brief summary
Rarely (~ once a month), the system become extremely slow, and mouse/keyboard are extremly laggy.

Looking at CPU consumption it come from the usb qube ("sys-usb" in my case), consuming over 50% of total CPU (Probably eating everything it is allowed to).
When I look at the process list, the issue is "kworker/*:*events".
The journal logs are filled with the same message: "hid-generic XXXXXXX: control queue full"

My "sys-usb" qube is a static disposableVM based on Debian 10 ( and currently using kernel 5.6.11-1 but it is not related to kernel version since I got the same issue months ago when this kernel version didn't existed yet).
The issue is not related to plugging or unplugging USB devices.

To Reproduce
No idea

Solutions you've tried
Restarting the "sys-usb" qube. Problem is gone after that.

Relevant documentation you've consulted
None
Found this error referenced on the archlinux documentation: https://wiki.archlinux.org/index.php/APC_UPS
And on gentoo forum: https://forums.gentoo.org/viewtopic-p-7569028.html

Related, non-duplicate issues
None

[h01ger]

On Thu, May 14, 2020 at 09:04:31AM -0700, Yukikoo wrote: **Brief summary** Rarely (~ once a month), the system become extremely slow, and mouse/keyboard are extremly laggy. Looking at CPU consumption it come from the usb qube ("sys-usb" in my case), consuming over 50% of total CPU (Probably eating everything it is allowed to). When I look at the process list, the issue is "kworker/*:*events".

I see this issue too. Thanks for reporting it!

…

-- cheers, Holger

------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

[marmarek]

I've seen it few times (definitely rarer than once a month), and re-plugging HID devices (keyboard/mouse etc) fixed it, without restarting sys-usb. But I don't have idea (yet) why it happens.

[h01ger]

On Thu, May 14, 2020 at 04:16:55PM -0700, Marek Marczykowski-Górecki wrote: re-plugging HID devices (keyboard/mouse etc) fixed it, without restarting sys-usb.

thank you very much for this workaround, I've just used it successfully! :)

…

-- cheers, Holger

------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

[SvenSemmler]

I see a similar issue. Re-plugging the HID devices does not help. However, I know what causes it in my case for sure: it happens whenever I connect my 2 TB external USB3 drive (backup). If I disconnect it the problem goes away. If I reconnect it, it comes back. The drive is encrypted. As soon as I entered the passphrase to mount it, the issue becomes less severe but doesn't disappear.

Let me know if there are any logs I should share.

[neowutran]

Do you confirm that this is the same issue ?
Aka: "hid-generic XXXXXXX: control queue full" spammed in the system logs (dmesg)

If it is the same issue, and that you can reproduce the issue at will, maybe you could try something like this ? https://wiki.archlinux.org/index.php/APC_UPS#hid-generic_:_control_queue_full ?
( using distributions kernel [https://www.qubes-os.org/doc/managing-vm-kernel/] )

[SvenSemmler]

I am sorry for the noise. I had the issue for many weeks and reproduced it 4-5 times before writing the post yesterday. Now I can no longer make it happen! This was either an issue with my external disk that has self-repaired or one of the many updates I installed yesterday evening.

[Liquidator2048]

hid-generic 0003:1532:0064.0003: control queue full
same issue using fedora-30 as template for sys-usb and kernel 4.19.107-1.

Solutions I've tried unplugging and plugging usb mouse and problem gone away for a while

[ptitdoc]

I have similar issues occurring quite often (CPU consumption). The reason for me most of the time is that sys-usb starts swapping causing high CPU (kworker is related the swapping in my case).
I have very low available RAM on sys-usb when the other Qubes consume all the RAM, and somehow sys-usb is not able to get enough RAM to work properly.

[ptitdoc]

I tried changing my USB-vm ram from 300 to 400MB. I probably need to use it several weeks before being sure that it is the problem.

My high memory usage may be because in my case I'm using some tools from within the sys-usb VM, because USB-passthrough does not work for all devices, and especially for mobile phones (probably because they are resetting USB or changing USB mode all the time, it disconnects the USB device from the attached VM before being able to use it).

[ptitdoc]

I suggest for those where it occurs to run top from dom0 to get more insight of the problem:
[dom0 ~]$ qvm-run --pass-io sys-usb 'top -b'

+1 i also have this issue and have had it sporadically for a while. f'ing heisenbug. its just infrequent enough that i havent bothered tracking it down properly until now. it is bad enough right now that i cant type faster than 1 char per second without it missing input characters. oterwie my typingloks lik this. i normally just reboot sys-usb to reboot, as physically getting to usb is a nuisance. can confirm it can be fixed temporarily by unplugging all usb things and seems to be fixed with removal of usb kbs and related hubs only (possibly important, does not correct just removing 'real kb', must also remove yubikey(s) and seemingly mouse, so all hid devices?) and re-plugging them in. it seems to happen somewhat regularly when i leave my terminal alone for many hours. i did a full re-install (on a different network, with a different m2 drive) for unrelated reasons - and the fresh install without my backup qubes being restored has same issue 'out of the box'. removing usb kvm switch and hubs does not fix issue - aka. same issue without usb hubs. also using debian as default vm. when this occurs, only noticeable behavioral problem is slow input, everything else is fine. can sometimes create a false problem at login screen, as the typing at normal speed without realizing issue is active misses characters of password without really noticing. until i clued into this, i thought i must have just forgotten my hard to remember long pw and was super mad at self.

fyi:
top in dom0 only shows 5-6% cpu usage.

in sys-usb it chews (kworker /1:2-events) 90+ cpu and is spikey. Also in sys-usb top MiB total Mem shows 220.4 total and 3.9 free. swap has ~900 free in all cases so i dont think it's a swap cpu issue for me.

dmesg is full of hid-generic xxxx:xxxx:xxxx.0009: control queue full msgs. i'm not particularly technical, but if i'm reading the log right, there are roughly around 100 control queue full messages in a burst, every 200,000 micro seconds (eg. message 0 is [xxxxx.023595], message ~99 is [xxxxx.028887], then message ~100 is [xxxxx.223171], message ~199 is [xxxxx.228899], message ~200 is [xxxx.420379], message ~299 is [xxxxx.425594].

also: after disconnecting things one at a time and going through many pages of messages, i think it might be one specific target device. when disconnecting i get
[time] usb x-y: USB disconnect, device number z
[time] hid-generic xxxx:xxxx:xxxx.0009: usb_submit_urb(ctrl) failed: -19

then the control queue full messages stop.

Other notes: I just unplugged and plugged back in all usb devices without a sys-usb reboot and it's back to normal. Xorg is leading cpu usage with 0.3%. free mem: 6.0. swap free: 908.5

im going to try to correct by increasing sys-usb to 400mb. i'll post in a while (possibly days/weeks) if it does or doesnt come back.