Repository navigation
Closed
Labels
bugimpacts-release-2.8.2Issues reported against NUT release 2.8.2 (maybe vanilla or with minor packaging tweaks)Issues reported against NUT release 2.8.2 (maybe vanilla or with minor packaging tweaks)service/daemon start/stopGeneral subject for starting and stopping NUT daemons (drivers, server, monitor); also BG/FG/DebugGeneral subject for starting and stopping NUT daemons (drivers, server, monitor); also BG/FG/Debug
Milestone
Description
The room was noisy and warm. PC investigation led to some discoveries:
strace shows intensive looping (about 15k lines per second) for what seems to be empty reads of a bit too many sockets (e.g. some detached wrongly so zombying around?) while log bumped to verbosity 6 does not show anything about these polls and is not that intensive (just detailed usb data updates, a few screens per sec)
select(16, [9 10 11 12 13 14 15], NULL, NULL, {tv_sec=0, tv_usec=783537}) = 5 (in [11 12 13 14 15], left {tv_sec=0, tv_usec=783536})
read(15, "", 512) = 0
read(14, "", 512) = 0
read(13, "", 512) = 0
read(12, "", 512) = 0
read(11, "", 512) = 0
select(16, [9 10 11 12 13 14 15], NULL, NULL, {tv_sec=0, tv_usec=783468}) = 5 (in [11 12 13 14 15], left {tv_sec=0, tv_usec=783467})
read(15, "", 512) = 0
read(14, "", 512) = 0
read(13, "", 512) = 0
read(12, "", 512) = 0
read(11, "", 512) = 0
select(16, [9 10 11 12 13 14 15], NULL, NULL, {tv_sec=0, tv_usec=783399}) = 5 (in [11 12 13 14 15], left {tv_sec=0, tv_usec=783398})
read(15, "", 512) = 0
read(14, "", 512) = 0
^Cstrace: Process 2232101 detached
<detached ...>
root@pve:~# ls -la /proc/2232101/fd
total 0
dr-x------ 2 root root 0 Jun 19 23:15 .
dr-xr-xr-x 9 nut nut 0 Apr 19 16:13 ..
lr-x------ 1 root root 64 Jun 19 23:15 0 -> /dev/null
lrwx------ 1 root root 64 Jun 19 23:15 1 -> 'socket:[608865694]'
lrwx------ 1 root root 64 Jun 19 23:15 10 -> 'socket:[608870464]'
lrwx------ 1 root root 64 Jun 19 23:15 11 -> 'socket:[608877633]'
lrwx------ 1 root root 64 Jun 19 23:15 12 -> 'socket:[608880779]'
lrwx------ 1 root root 64 Jun 19 23:15 13 -> 'socket:[608880826]'
lrwx------ 1 root root 64 Jun 19 23:15 14 -> 'socket:[608985678]'
lrwx------ 1 root root 64 Jun 19 23:15 15 -> 'socket:[608985715]'
lrwx------ 1 root root 64 Jun 19 23:15 2 -> 'socket:[608865694]'
lrwx------ 1 root root 64 Jun 19 23:15 3 -> 'socket:[608864985]'
lrwx------ 1 root root 64 Jun 19 23:15 4 -> 'socket:[608864991]'
lrwx------ 1 root root 64 Jun 19 23:15 5 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Jun 19 23:15 6 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Jun 19 23:15 7 -> 'anon_inode:[timerfd]'
lrwx------ 1 root root 64 Jun 19 23:15 8 -> /dev/bus/usb/003/002
lrwx------ 1 root root 64 Jun 19 23:15 9 -> 'socket:[608864012]'
top excerpt:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2232101 nut 20 0 16412 4108 3536 R 93.8 0.0 48437:18 usbhid-ups
CPU Load over the 2 months it runs is consistently around 1 core always busy.


# systemctl status nut-driver@eco650 -l | cat
● nut-driver@eco650.service - Network UPS Tools - device driver for NUT device 'eco650'
Loaded: loaded (/lib/systemd/system/nut-driver@.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/nut-driver@eco650.service.d
└─nut-driver-enumerator-generated-checksum.conf, nut-driver-enumerator-generated-devicename.conf, nut-driver-enumerator-generated.conf
Active: active (running) since Fri 2024-04-19 16:13:53 CEST; 2 months 0 days ago
Main PID: 2232101 (usbhid-ups)
Tasks: 2 (limit: 154419)
Memory: 1.1M
CPU: 1month 3d 4h 48min 41.608s
CGroup: /system.slice/system-nut\x2ddriver.slice/nut-driver@eco650.service
└─2232101 /usr/local/ups/bin/usbhid-ups -FF -a eco650
Warning: journal has been rotated since unit was started, output may be incomplete.
Debugging is not enabled (was prime suspect):
:; upsc eco650
battery.charge: 100
battery.charge.low: 20
battery.runtime: 1268
battery.type: PbAc
device.mfr: EATON
device.model: Ellipse ECO 650
device.serial: 000000000
device.type: ups
driver.debug: 0
driver.flag.allow_killpower: 0
driver.name: usbhid-ups
driver.parameter.bus: 003
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.product: Ellipse ECO
driver.parameter.productid: FFFF
driver.parameter.serial: 000000000
driver.parameter.synchronous: auto
driver.parameter.vendor: EATON
driver.parameter.vendorid: 0463
driver.state: quiet
driver.version: 2.8.2-113-g3d3f41bbe
driver.version.data: MGE HID 1.46
driver.version.internal: 0.53
driver.version.usb: libusb-1.0.24 (API: 0x1000108)
input.transfer.high: 264
input.transfer.low: 161
outlet.1.desc: PowerShare Outlet 1
outlet.1.id: 2
outlet.1.status: on
outlet.1.switchable: yes
outlet.2.desc: PowerShare Outlet 2
outlet.2.id: 3
outlet.2.status: on
outlet.2.switchable: yes
outlet.desc: Main Outlet
outlet.id: 1
outlet.power: 25
outlet.switchable: no
output.frequency.nominal: 50
output.voltage: 230.0
output.voltage.nominal: 230
ups.beeper.status: disabled
ups.delay.shutdown: 20
ups.delay.start: 30
ups.firmware: 02
ups.load: 19
ups.mfr: EATON
ups.model: Ellipse ECO 650
ups.power.nominal: 650
ups.productid: ffff
ups.realpower: 99
ups.serial: 000000000
ups.status: OL
ups.timer.shutdown: -1
ups.timer.start: -1
ups.vendorid: 0463
Metadata
Metadata
Assignees
Labels
bugimpacts-release-2.8.2Issues reported against NUT release 2.8.2 (maybe vanilla or with minor packaging tweaks)Issues reported against NUT release 2.8.2 (maybe vanilla or with minor packaging tweaks)service/daemon start/stopGeneral subject for starting and stopping NUT daemons (drivers, server, monitor); also BG/FG/DebugGeneral subject for starting and stopping NUT daemons (drivers, server, monitor); also BG/FG/Debug
Activity
jimklimov commentedon Jun 20, 2024
I've heard the debugger start! The traced process was stalled for a minute, and CPU fans went off...
jimklimov commentedon Jun 20, 2024
Seems indeed like a socket closet on the other side:
Checking if this can be a case of a writer preparing their data but the buffer (kernel?) not yet ready to spit it out, especially when CPU is too busy to work due to this tight cycle (per one of those discussions).
Commit 3d3f41b is from March and there were no changes to
dstate.csince. So the bug is part of at least NUT v2.8.2 release.Got a rough guess then that this is a fallout of work on extending Unix socket protocol to send signals (reload etc.) - if a sibling driver process (or
upsdrvctl) connects to the socket file, spews "go reload" and exits - and the running daemon does not close such sockets, it would add such "is ready/got nothing" sockets and tight polling loops. If this is the case, probably the bug is there since forever, and restartingupsdwhile a driver is alive would reproduce it too.The new feature #1903 (et al) just exposes it more (old lifecycle of the daemon did not deal with many socket connections/disconnections), and probably
upsddoes have ways around that since it deals with a population of drivers coming and going (or also has the bug?..)jimklimov commentedon Jun 20, 2024
Indeed, sending a reload command via socket increases the open file descriptor count on the driver daemon:
jimklimov commentedon Jun 20, 2024
And
upsddoes not suffer this situation, killing a driver closes the server-side file descriptor:UPDATE: also checked with
kill -9so the driver has no chance to say it is exiting.jimklimov commentedon Jun 20, 2024
For a driver is easy to reproduce, using the
-c reloadfeature from #1906 :...so an eventual fix would be easy to check.
jimklimov commentedon Jun 20, 2024
Hm,
O_NDELAYvs.O_NONBLOCK: gotta check which one we use (the former is per older standards (pre-1988!) and causesread()to return 0, indiscernible from EOF; the latter is more informative): https://comp.unix.programmer.narkive.com/cFN0PRBQ/select-rtns-ready-read-returns-0#post12jimklimov commentedon Jun 20, 2024
Oh...
jimklimov commentedon Jun 20, 2024
As some posts say, on many systems the two definitions are identical. In practice, quick replacement in
dstate.candupsdrvquery.cdid not help (the latter actually checks forif (upsdrvquery_read_timeout(conn, tv) < 1) goto socket_error;cases.The
upssched.coffers an example of dealing with this already:nut/clients/upssched.c
Lines 687 to 741 in ab9d694
drivers/dstate.c, drivers/upsdrvquery.c: err on the safer side and se…
drivers/dstate.c, NEWS.adoc: a read() returning 0 after a select with…
drivers/dstate.{c,h}: introduce conn_t->readzero field and throttles …