[FreeBSD] psutil.pids() fails with [Errno 12] Cannot allocate memory (originated from sysctl)

[pilot513]

Summary

• OS: { FreeBSD 13.0 P2 }
• OS Core: 13.0-RELEASE-p2 FreeBSD 13.0-RELEASE-p2 Test suite failures on Linux #2 releng/13.0-n244746-b74cdf1ecea
• Architecture: { 64bit, Intel(R) Xeon(R) CPU E5630 @ 2.53GHz }
• Psutil version: { py38-psutil-5.9.0 }
• Python version: { Python 3.8.13 }
• Type: core

Description

04/04/22 (23:39:26) | ERROR: Data collection thread failed with exception: [Errno 12] Cannot allocate memory (originated from sysctl)
Traceback (most recent call last):
  File "/usr/local/bin/bpytop", line 2958, in _runner
    collector._collect()
  File "/usr/local/bin/bpytop", line 3728, in _collect
    for p in sorted(psutil.process_iter(cls.p_values + (["memory_info"] if CONFIG.proc_mem_bytes else []), err), key=lambda p: eval(sort_cmd), reverse=reverse):
  File "/usr/local/lib/python3.8/site-packages/psutil/__init__.py", line 1432, in process_iter
    a = set(pids())
  File "/usr/local/lib/python3.8/site-packages/psutil/__init__.py", line 1372, in pids
    ret = sorted(_psplatform.pids())
  File "/usr/local/lib/python3.8/site-packages/psutil/_psbsd.py", line 512, in pids
    ret = cext.pids()
OSError: [Errno 12] Cannot allocate memory (originated from sysctl)
04/04/22 (23:39:28) | WARNING: Exiting with errorcode (1). Runtime 13:47:42

[giampaolo]

Ouch! This is bad. The error originates from here.

The fix should consist in calling the first sysctl() in a loop, and enlarge the buffer size if it returns ENOMEM. There are similar constructs elsewhere in psutil. I'm CCing @glebius and @kostikbel just in case they have some comments about this.

Since I've never seen this happening here's a question: how many processes are running on this machine? ps aux | wc -l should tell.

[giampaolo]

Also how often does this happen? Can you reproduce it reliably?

[pilot513]

It's show working bashtop now. Not many processes. Basically it's a few virtual servers in VirtualBox

[pilot513]

$ uptime
20:17 up 2 days, 1:37, 4 users, load averages: 0,32 0,38 0,34

[pilot513]

This happened for the second time in two days. Therefore, I decided to write, because thought it would be important.
Yes, I can run both applications in parallel: bpytop / bashtop
Let's see what happens.

[pilot513]

Ok. I'm run that:

Waiting now some time

[pilot513]

So far, the apps are running smoothly. Let's see what will happen next.

[frikilax]

Hello there!
Just wanted to say that this also happens on HardenedBSD 12.3 (FreeBSD variant)!

uname -a : FreeBSD hostname 12.3-STABLE-HBSD FreeBSD 12.3-STABLE-HBSD #0 : Mon Apr 18 00:48:16 UTC 2022 root@ci-12.md.hardenedbsd.lan:/usr/obj/usr/src/amd64.amd64/sys/HARDENEDBSD amd64
python version : 3.8.13
psutil version : 5.9.1

I rollbacked to psutil 5.8.0 and the error doesn't seem to happen anymore

[giampaolo]

This should now be fixed by c7c7bbc which dynamically increase the buffer size instead of immediately giving up with ENOMEM.
Please confirm if you have the chance.

[frikilax]

I can confirm this resolved my issue, thanks!