USB input devices do not work unless they are replugged

[nl6720]

systemd version the issue has been seen with

254rc3-1

Used distribution

Arch Linux

Linux kernel version used

6.4.5-arch1-1

CPU architectures issue was seen on

x86_64

Component

systemd-udevd

Expected behaviour you didn't see

USB devices plugged into the monitor's USB hub work in a plasma wayland session.

Unexpected behaviour you saw

After updating systemd to 254rc2-2, USB devices attached to my monitor's USB hub do not work in Plasma wayland session unless I unplug and replug them (or the USB/Thunderbolt3 cable connecting the monitor to the PC).
The USB devices work fine in a Plasma Xorg session and also in LightDM+web-greeter which uses Xorg.

From what I can tell, rc1 is bad too.

See also Arch bug FS#79144.

Steps to reproduce the problem

Start a plasmawayland session.

Additional program output to the terminal or log subsystem illustrating the issue

No response

[nl6720]

Some logs:

• systemd-253.6-2.plasma_wayland.good.log
• systemd-254rc2-2.plasma_wayland.bad.log
  In systemd-254rc2-2.plasma_wayland.bad.log I unplugged the keyboard at 12:23:06.
• systemd-254rc2-2.plasma_xorg.good.log
• udevadm_info.253.6-2.good.log
• udevadm_info.254rc2-2.replug.log
• udevadm_info.254rc2-2.bad.log

[yuwata]

Maybe duplicate of #24026 ??
Instead of replugging USB devices, what happens when wayland or anything graphical things started slightly later?
E.g. boot with CLI mode (boot with 3 kernel command line option), then switch to GUI mode (login CLI console and call systemd isolate graphical.target) ?
Or, insert a delay before starting graphical things, e.g. adding the following dropin that starts wayland or plasma (I am not familiar which service should have the following though):

# /etc/systemd/system/foo.service.d/override.conf <-- please replace 'foo'
[Service]
ExecStartPre=sleep 10

[nl6720]

I'm using LightDM with web-greeter which do not use wayland. The unit is lightdm.service. But the USB devices (keyboard and mouse, in my case) work fine in the display manager. Only when I start the plasma wayland session from the display manager, do things go wrong.

[nl6720]

Nothing changed after adding the drop-in for lightdm.service and booting with systemd.unit=multi-user.target + running systemd isolate graphical.target from the tty.

[yuwata]

Thanks. Then, that's not a duplicate of #24026. Hmm...

Is the issue only happens when keyboard or mouse are connected to the USB hub? What happens if they are connected to the machine's USB port?

[yuwata]

Though I am not familiar with plasma or wayland, but I cannot find anything spurious in the provided logs.
Could you make them more verbose?

Also, is the key of the issue is in wayland or not? What happens when you use Gnome or any others?

[nl6720]

The same issue occurs when the mouse & keyboard are connected to the motherboard's USB ports. Oops, I could have sworn I tested it before, I guess I forgot. And it looks like it only happens with input devices(?), as a USB flash drive is accessible just fine.

Starting a plasma session from the tty with startplasma-wayland doesn't exhibit this issue. 🤔

Booting with systemd.log_level=debug gave me strange results. The issue only appeared the third time I logged in from LightDM. 1.log

[nl6720]

The same issue occurs when the mouse & keyboard are connected to the motherboard's USB ports. Oops, I could have sworn I tested it before, I guess I forgot.

Trying it over and over again, with the mouse connected to the front USB port, sometimes it works while the keyboard connected via the monitor doesn't. I tried using SDDM and I couldn't trigger the issue. This is annoying. ☹️

So... something in systemd v254rc1+ broke something USB input related for plasma-wayland sessions started via LightDM?!

Switching from LightDM to SDDM is not an option for me because I need multi-seat support.

[yuwata]

One spurious(?) thing I found is that the following symlinks are requested by two devices, and the owner is swapped on replugging.

/dev/input/by-id/usb-Logitech_USB_Keyboard-event-if01
/dev/input/by-path/pci-0000:12:00.0-usb-0:2.1:1.1-event
/dev/input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event

$ grep input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event 1.log
Jul 25 14:30:43 oak (udev-worker)[1000]: event5: /usr/lib/udev/rules.d/60-persistent-input.rules:44 Added SYMLINK 'input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event'
Jul 25 14:30:43 oak (udev-worker)[1000]: event5: Successfully created symlink '/dev/input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event' to '/dev/input/event5'
Jul 25 14:30:43 oak (udev-worker)[1005]: event6: /usr/lib/udev/rules.d/60-persistent-input.rules:44 Added SYMLINK 'input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event'
(unplug)
Jul 25 14:32:47 oak (udev-worker)[4153]: event5: Successfully created symlink '/dev/input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event' to '/dev/input/event6'
Jul 25 14:32:47 oak (udev-worker)[4153]: event6: No reference left for '/dev/input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event', removing
(replug)
Jul 25 14:32:51 oak (udev-worker)[4180]: event6: /usr/lib/udev/rules.d/60-persistent-input.rules:44 Added SYMLINK 'input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event'
Jul 25 14:32:51 oak (udev-worker)[4180]: event6: Successfully created symlink '/dev/input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event' to '/dev/input/event6'
Jul 25 14:32:51 oak (udev-worker)[4177]: event5: /usr/lib/udev/rules.d/60-persistent-input.rules:44 Added SYMLINK 'input/by-path/pci-0000:12:00.0-usbv2-0:2.1:1.1-event'

As you can see, initially the link owned by event5, but later event6.

And, it seems event6 is something important, at least it is managed by logind.

$ grep systemd-logind.*event[56] 1.log
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event5 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Watching system buttons on /dev/input/event6 (Logitech USB Keyboard System Control)
(unplug+replug)
Jul 25 14:32:51 oak systemd-logind[1374]: Watching system buttons on /dev/input/event6 (Logitech USB Keyboard System Control)
Jul 25 14:32:51 oak systemd-logind[1374]: Device /dev/input/event5 does not expose keys or switches relevant to us, ignoring.

But I do not know if the swapping of by-id or by-path symlinks is essential.
As the mouse devices have unique symlinks, and there is another keyboard related device.

$ grep 'S: input/by-id' udevadm_info.254rc2-2.bad.log 
                          |     | : S: input/by-id/usb-Logitech_USB_Keyboard-event-kbd
                          |   |   : S: input/by-id/usb-Logitech_USB_Keyboard-event-if01
                          |       : S: input/by-id/usb-Logitech_USB_Keyboard-event-if01
                          | | | | | : S: input/by-id/usb-Logitech_USB_Receiver-if02-event-mouse
                          | | | |   : S: input/by-id/usb-Logitech_USB_Receiver-if02-mouse

$ grep systemd-logind.*input/event[0-9]* 1.log
Jul 25 14:30:44 oak systemd-logind[1374]: Watching system buttons on /dev/input/event1 (Power Button)
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event2 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Watching system buttons on /dev/input/event0 (Power Button)
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event13 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event14 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event15 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event16 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Watching system buttons on /dev/input/event3 (Logitech USB Keyboard)
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event5 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Watching system buttons on /dev/input/event6 (Logitech USB Keyboard System Control)
Jul 25 14:30:44 oak systemd-logind[1374]: Watching system buttons on /dev/input/event7 (Logitech MX Master 3)
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event10 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event11 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event12 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event17 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event18 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event19 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event20 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event21 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:30:44 oak systemd-logind[1374]: Device /dev/input/event9 does not expose keys or switches relevant to us, ignoring.
Jul 25 14:32:51 oak systemd-logind[1374]: Watching system buttons on /dev/input/event3 (Logitech USB Keyboard)
Jul 25 14:32:51 oak systemd-logind[1374]: Watching system buttons on /dev/input/event6 (Logitech USB Keyboard System Control)
Jul 25 14:32:51 oak systemd-logind[1374]: Device /dev/input/event5 does not expose keys or switches relevant to us, ignoring.

Also, such conflicts of symlinks can be also observerd in v253...

[yuwata]

@nl6720 BTW, is it possible to reproduce the issue with LightDM+plasma+wayland without multi-seat setup?