[Linux] cpu_percent() and virtual_memory() may yield an exception due to internal errors

[andreasbrueg]

Summary

• OS: Linux 5.4.78-2-pve (Proxmox)
• Architecture: x86_64
• Psutil version: 5.8.0
• Python version: Python 3.7.3
• Type: Core

Description

Calling psutil.cpu_percent() may lead to an exception. I am calling this method in some separate thread to monitor system performance. Most of the time, calling the method works as expected. In some rare cases, it does not and yields an exception. I cannot see any pattern in when it crashes and when it does not. In general, rerunning the same program that crashed before does likely not yield an exception when running again. Crashes are difficult to reproduce due to their rarity. At the same time, for some long running benchmarks they can appear in quick succession.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "[own program].py", line 35, in logger_thread
    cpu_p = psutil.cpu_percent()
  File "/usr/local/lib/python3.7/dist-packages/psutil/__init__.py", line 1747, in cpu_percent
    _last_cpu_times = cpu_times()
  File "/usr/local/lib/python3.7/dist-packages/psutil/__init__.py", line 1607, in cpu_times
    return _psplatform.cpu_times()
  File "/usr/local/lib/python3.7/dist-packages/psutil/_pslinux.py", line 607, in cpu_times
    return scputimes(*fields)
TypeError: __new__() missing 10 required positional arguments: 'user', 'nice', 'system', 'idle', 'iowait', 'irq', 'softirq', 'steal', 'guest', and
 'guest_nice'

I also got some rare cases where psutil.virtual_memory() leads to an exception:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "[own program].py", line 36, in logger_thread
    ram = psutil.virtual_memory()
  File "/usr/local/lib/python3.7/dist-packages/psutil/__init__.py", line 1963, in virtual_memory
    ret = _psplatform.virtual_memory()
  File "/usr/local/lib/python3.7/dist-packages/psutil/_pslinux.py", line 440, in virtual_memory
    total = mems[b'MemTotal:']
KeyError: b'MemTotal:'

[giampaolo]

Very weird bug. It looks like /proc/stat or /proc/meminfo are empty for some reason. The fact that it happens sporadically is even more weird. Are you sure you're not doing something nasty in your code... like monkey patching open() builtin or something?