process.children(recursive=True) could rise Type error

[deploy-adbeat]

  File "worker_health_monitoring.py", line 30, in get_processes_data
    children_processes = process.children(recursive=True)
  File "~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/psutil/__init__.py", line 343, in wrapper
    return fun(self, *args, **kwargs)
  File "~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/psutil/__init__.py", line 964, in children
    for p in process_iter():
  File "~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/psutil/__init__.py", line 1484, in process_iter
    list(dict.fromkeys(new_pids).items())):
TypeError: '<' not supported between instances of 'NoneType' and 'Process'

Python version 3.6.6. psutil version: 5.2.2
Probably some of the new_pids returned None instead of Process

[giampaolo]

Do you have a way to reproduce the issue? Can you print _pmap and new_pids before the exception occurs?

[deploy-adbeat]

It's very unfrequent, one to many thousands calls. I think it's related to situation when one of the processes ends during method execution. It exists in the begging, end does not sometimes later.

[giampaolo]

Looking back at this... it's a nasty one. It appears it's a race condition. Are you using process_iter() from 2 different threads? process_iter uses a global variable so I suspect it could be related with thread-safety.

[giampaolo]

Despite I couldn't reproduce this issue I assume 96091c2 fixed it. Closing this out.

[JMKS87]

Hi, I've encountered this Exception twice today, once with 5.6.1 (this is understandable), but the 2nd time it was after updating to 5.6.7.
It is as described in this topic, in Threaded environment, under a Flask app.
Although I don't suspect that process_iter is used twice at the same time, I couldn't be sure, though; can happen if it will iterate for unusually long time, like 10mins.

I can provide Sentry stacktrace & more info if that could help.

[giampaolo]

:-(
How are you using process_iter() exactly? Are you using the returning iterator as a global object? Try showing some code.

[JMKS87]

Thanks for the reply, I'll provide some more insight as I see my example.

I had it happen 2 more times today (for about 350 parent functions calls total).

process_iter is used in 2 places, on separate functions (so - local usage, definitely). Both these functions are called in separate Threads and could run simultanously, because all calls are making new Threads, instead of serial processing.

I tried to reproduce it locally, and was unable to as for now.

Some code snippets & traceback below.

Sample traceback:

Traceback (most recent call last):
  File "/app/src/config/exceptions.py", line 35, in run_with_except_hook
    run_old(*args, **kw)
  File "/usr/local/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/app/src/helpers/psutil_tools.py", line 73, in close_excess_processes
    for proc in psutil.process_iter():
  File "/usr/local/lib/python3.7/site-packages/psutil/__init__.py", line 1560, in process_iter
    list(dict.fromkeys(new_pids).items()))
TypeError: '<' not supported between instances of 'NoneType' and 'Process'"   
  python_logger: "src.config.exceptions"   

I tried to include everything that would make it as close to production environment and complexity as possible
Code sample:
https://hastebin.com/akiwulohig.py
(excessive code here and there, but I hope it's readable)

[lukeboyes-serato]

Hi there,

We also get this in our production environment ~2 times a week and also use it from multiple threads.

File "atvirtualenv3/lib/python3.7/site-packages/psutil/__init__.py", line 1434, in process_iter list(dict.fromkeys(new_pids).items())) TypeError: '<' not supported between instances of 'NoneType' and 'Process'

I wonder if moving the lock to wrap this whole code block would work?

psutil/psutil/__init__.py

Lines 1426 to 1435 in ea5b2df

	a = set(pids())
	b = set(_pmap.keys())
	new_pids = a - b
	gone_pids = b - a
	for pid in gone_pids:
	remove(pid)
	with _lock:
	ls = sorted(list(_pmap.items()) +
	list(dict.fromkeys(new_pids).items()))

[giampaolo]

This was fixed a while ago, process_iter() is thread safe now. The ticket just never got closed.