usbguard stops responding when recvmsg receives ENOBUFS

[jdstrand]

Hi,

I just filed this bug in Ubuntu: https://launchpad.net/bugs/1855189. This is reported against 0.7.4 and 0.7.5. 0.7.6 doesn't seem to have changed anything in this area and is presumably affected (as are earlier versions). I'll further mention that while the reproducer appears contrived, Ubuntu users are seeing the ENOBUFS error occasionally with real world usage.

"
With 0.7.4+ds-1 from 19.10, usbguard may stop responding to events when recvmsg fails with ENOBUFS. To reproduce:

  while /bin/true ; do
    sudo udevadm control --reload-rules
    sudo udevadm trigger
    sudo udevadm settle --timeout=3
  done

Eventually, this pops out in the journald logs:

  usbguard-daemon[12488]: ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
  usbguard-daemon[12488]: UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available

and usbguard will no longer process events until restarted (sudo systemctl restart usbguard).

I then installed usbguard 0.7.5 from the desktop team's PPA[1], ran the loop and saw the equivalent error pop out:

  usbguard-daemon[5958]: [1575487887.438] (E) ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
  usbguard-daemon[5958]: [1575487887.438] (E) UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available

This is coming from https://github.com/USBGuard/usbguard/blob/master/src/Library/UEventDeviceManager.cpp#L528 where recvmsg() is returning -1 with 'No buffer space available' (ENOBUFS). Looking at sysdeps/gnu/errlist.c in glibc, ENOBUFS is documented as:

  #ifdef ENOBUFS
  /*
  TRANS The kernel's buffers for I/O operations are all in use.  In GNU, this
  TRANS error is always synonymous with @code{ENOMEM}; you may get one or the
  TRANS other from network operations. */
      [ERR_REMAP (ENOBUFS)] = N_("No buffer space available"),

It seems that usbguard should be able to handle transient ENOBUFS scenarios with uevent storms and recover when the kernel's buffers are back in order. In my testing, I treated ENOBUFS similarly to EAGAIN/EWOULDBLOCK by logging a warning and simply returning[2]:

  usbguard-daemon[2405]: ueventProcessRead: failed to read pending uevent (returning): rc=-1 errno=105

It appears usbguard continues to function (eg, if I do the loop for a long time, usbguard remains responsive and processes other uevents, but I'm not sure this is the correct approach since, for example, UEventDeviceManager::ueventOpen() speaks to this error condition by saying: "Set a 1MiB receive buffer on the netlink socket to avoid ENOBUFS error in recvmsg").

[1]https://launchpad.net/~ubuntu-desktop/+archive/ubuntu/usbguard
[2]exploratory patch (do not commit in Ubuntu without upstream confirmation)

Index: usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
===================================================================
--- usbguard-0.7.4+ds.orig/src/Library/UEventDeviceManager.cpp
+++ usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
@@ -468,6 +468,12 @@ namespace usbguard
           << "reading from uevent source would block thread execution";
         return;
       }
+      else if (saved_errno == ENOBUFS) {
+        USBGUARD_LOG(Error) << "ueventProcessRead: "
+          << "failed to read pending uevent (returning): "
+          << "rc=" << rc << " errno=" << saved_errno;
+        return;
+      }
       else {
         USBGUARD_LOG(Error) << "ueventProcessRead: "
           << "failed to read pending uevent: "

"

[superm1]

@jdstrand by chance have you tried with 0.7.6 yet as well? I wonder if the changes in a8046a2 to help the race conditions from uevents settling might also help the situation that was leading to uevent storms leading to -ENOBUFS

[seb128]

@superm1 it's still happening with 0;7.6 on Ubuntu focal, using the udevadm calls in a small loop of 10 iterations isn't enough to trigger the error almost every time on a recent xps13 (the while true loop from @jdstrand makes the computer unresponsive though)

[rathann]

This happens on Fedora 31 as well: https://bugzilla.redhat.com/show_bug.cgi?id=1789923 .