Pulse monitoring causes OpenBSD 7.6 VM to freeze

[F-Stop]

Describe the bug
Pulse's monitoring of two OpenBSD 7.6 VMs running in ProxMox 8.4.14 caused the VMs to crash. After an overnight run (original text: "less than an hour of polling"), the VMs were unable to be SSH'd into and became unresponsive to the QEMU guest agent (the VMs had to be hard shut down). After restart one of the VMs crashed again after a bit of polling.

To Reproduce
Steps to reproduce the behavior:

1. Install OpenBSD 7.6 in a VM and have it run flawlessly for months
2. Install Pulse in an LXC and set it up to monitor Prox using the default scripts
3. Realize after an hour that you can no longer SSH into your OpenBSD vms

Expected behavior
To let my VMs run without crashing :)

Environment:

• Pulse Version: v4.28.0
• Installation Type: ProxmoxVE LXC (Ubuntu 24.04.3 LTS base)
• OpenBSD: 7.6
• QEMU Guest Agent (in OpenBSD): 9.0.2

Additional context
Logs on the OpenBSD VM revealed the following chain of errors in /var/log/messages (with the failed to read the two files repeating a lot before the process table got full):

Nov 11 12:39:02 servername qemu-ga: info: failed to read '/etc/os-release', error: Failed to open file M-bM^@M^\/etc/os-releaseM-bM^@M^]: No such file or directory
Nov 11 12:39:02 servername qemu-ga: info: failed to read '/usr/lib/os-release', error: Failed to open file M-bM^@M^\/usr/lib/os-releaseM-bM^@M^]: No such file or directory
Nov 11 12:39:02 servername qemu-ga: info: guest-ping called
Nov 11 12:39:42 servername last message repeated 3 times
Nov 11 12:41:02 servername last message repeated 4 times
Nov 11 12:41:02 servername /bsd: process: table is full
Nov 11 12:41:33 servername last message repeated 2 times
Nov 11 12:41:33 servername qemu-ga: info: guest-ping called
Nov 11 12:41:52 servername last message repeated 2 times
Nov 11 12:41:52 servername /bsd: process: table is full
Nov 11 12:42:12 servername qemu-ga: info: guest-ping called
Nov 11 12:42:12 servername /bsd: process: table is full

(edited to clarify the timeline)

[rcourtman]

Current evidence points to the QEMU guest agent on OpenBSD 7.6 launching helper processes whenever anything calls guest-get-osinfo, and those helpers keep trying to open /etc/os-release even though that file does not exist on OpenBSD. Pulse only triggers the RPC and the repeated messages such as guest ping or process table is full appear to originate inside qemu-ga or the kernel, but I have not found upstream documentation confirming this behavior yet.

Pulse already caches guest agent metadata for five minutes and enforces a per-VM refresh window, so each VM gets roughly twelve guest agent polls per hour even when calls fail. That rate should not come close to exhausting a process table by itself, which is why the investigation keeps pointing back to qemu-ga repeatedly spawning helpers that never exit.

It would help to capture a few details before the VM crashes. Please run sysctl kern.maxproc on the guest, then ulimit -u, to confirm the current process limits. If you have other tools or the Proxmox UI open that could be calling the guest agent, let me know so I can account for their traffic. Right before starting Pulse, run while true; do ps aux | wc -l; sleep 5; done in another shell and watch whether the count ramps up steadily when the monitor is polling.

As an immediate workaround, disable the guest agent inside that VM or upgrade to a newer OpenBSD guest agent build if one is available. Pulse will add detection for the unsupported OS info response and will skip that call for affected guests in the next release so the buggy code path does not keep firing.

[F-Stop]

Thanks for the super fast and detailed reply!

sysctl kern.maxproc reported "kern.maxproc=1310". ulimit is not on the system.

The process command helped a lot: it increases in count by 2 every 20 seconds. It started at 91, and slowly started ticking up. And, seeing this reminded me that I am an idiot, and forgot that I actually left the fresh install of Pulse running overnight before the BSD VM crash was observed. So, it is likely just slowly ticking up by 2 until it fails hours later (roughly 6.5 hours, if I have done my math right).

Spawned processes all look like this:

USER       PID %CPU %MEM   VSZ   RSS TT  STAT   STARTED       TIME COMMAND
root     92441  0.0  0.0     0     0 ??  Zp     -          0:00.00 (sh)
root     65096  0.0  0.0     0     0 ??  Zp     -          0:00.00 (sh)
root      9768  0.0  0.0     0     0 ??  Zp     -          0:00.00 (sh)
root      9435  0.0  0.0     0     0 ??  Zp     -          0:00.00 (sh)

[rcourtman]

Confirmed that qemu-ga 9.0.2 on OpenBSD 7.6 spawns zombie sh processes every time guest-get-osinfo runs, so the process count climbs by two every twenty seconds until kern.maxproc 1310 is exhausted.

Commit 3e90737 now tracks consecutive guest-get-osinfo failures per VM. After three failures Pulse stops calling guest-get-osinfo for that VM but keeps pulling the remaining guest agent data such as network interfaces and agent version. This avoids the buggy qemu-ga code path while preserving the rest of the data stream.

The failure counter resets on the next successful guest-get-osinfo call, so once qemu-ga is upgraded the OS info collection resumes automatically without manual steps.

Until then either disable the guest agent in the OpenBSD VM or remove that VM from Pulse monitoring to avoid the crash.

This mitigation will be in the next release.

[F-Stop]

I upgraded Pulse to 4.29.5, and also re-installed/configured the host monitoring agent in the ProxVE (did I need to do that after upgrading the Pulse server?).

Unfortunately, the same behavior is still occurring; the process count is increasing by 2 every 20 or so seconds in the OpenBSD virtual machine.

I am seeing the exact same error messages in the logs:

Nov 12 20:27:53 server qemu-ga: info: failed to read '/etc/os-release', error: Failed to open file M-bM^@M^\/etc/os-releaseM-bM^@M^]: No such file or directory
Nov 12 20:27:53 server qemu-ga: info: failed to read '/usr/lib/os-release', error: Failed to open file M-bM^@M^\/usr/lib/os-releaseM-bM^@M^]: No such file or directory
Nov 12 20:27:53 server qemu-ga: info: guest-ping called

[F-Stop]

FYI: I confirmed that the same behavior (increasing process count every 20 seconds) happens with QEMU guest agent v. 10.0.2 in a fresh OpenBSD 7.8 install in a VM in my ProxMox VE, using Pulse 4.29.5.