Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

process.children(recursive=True) could rise Type error #1353

Open
deploy-adbeat opened this issue Oct 22, 2018 · 8 comments
Open

process.children(recursive=True) could rise Type error #1353

deploy-adbeat opened this issue Oct 22, 2018 · 8 comments

Comments

@deploy-adbeat
Copy link

deploy-adbeat commented Oct 22, 2018

  File "worker_health_monitoring.py", line 30, in get_processes_data
    children_processes = process.children(recursive=True)
  File "~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/psutil/__init__.py", line 343, in wrapper
    return fun(self, *args, **kwargs)
  File "~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/psutil/__init__.py", line 964, in children
    for p in process_iter():
  File "~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/psutil/__init__.py", line 1484, in process_iter
    list(dict.fromkeys(new_pids).items())):
TypeError: '<' not supported between instances of 'NoneType' and 'Process'

Python version 3.6.6. psutil version: 5.2.2
Probably some of the new_pids returned None instead of Process

@giampaolo
Copy link
Owner

giampaolo commented Oct 23, 2018

Do you have a way to reproduce the issue? Can you print _pmap and new_pids before the exception occurs?

@deploy-adbeat
Copy link
Author

deploy-adbeat commented Oct 30, 2018

It's very unfrequent, one to many thousands calls. I think it's related to situation when one of the processes ends during method execution. It exists in the begging, end does not sometimes later.

@giampaolo
Copy link
Owner

giampaolo commented Feb 26, 2019

Looking back at this... it's a nasty one. It appears it's a race condition. Are you using process_iter() from 2 different threads? process_iter uses a global variable so I suspect it could be related with thread-safety.

@giampaolo
Copy link
Owner

giampaolo commented Feb 27, 2019

Despite I couldn't reproduce this issue I assume 96091c2 fixed it. Closing this out.

@JMKS87
Copy link

JMKS87 commented Jan 7, 2020

Hi, I've encountered this Exception twice today, once with 5.6.1 (this is understandable), but the 2nd time it was after updating to 5.6.7.
It is as described in this topic, in Threaded environment, under a Flask app.
Although I don't suspect that process_iter is used twice at the same time, I couldn't be sure, though; can happen if it will iterate for unusually long time, like 10mins.

I can provide Sentry stacktrace & more info if that could help.

@giampaolo
Copy link
Owner

giampaolo commented Jan 8, 2020

:-(
How are you using process_iter() exactly? Are you using the returning iterator as a global object? Try showing some code.

@giampaolo giampaolo reopened this Jan 8, 2020
@JMKS87
Copy link

JMKS87 commented Jan 8, 2020

Thanks for the reply, I'll provide some more insight as I see my example.

I had it happen 2 more times today (for about 350 parent functions calls total).

process_iter is used in 2 places, on separate functions (so - local usage, definitely). Both these functions are called in separate Threads and could run simultanously, because all calls are making new Threads, instead of serial processing.

I tried to reproduce it locally, and was unable to as for now.

Some code snippets & traceback below.

Sample traceback:

Traceback (most recent call last):
  File "/app/src/config/exceptions.py", line 35, in run_with_except_hook
    run_old(*args, **kw)
  File "/usr/local/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/app/src/helpers/psutil_tools.py", line 73, in close_excess_processes
    for proc in psutil.process_iter():
  File "/usr/local/lib/python3.7/site-packages/psutil/__init__.py", line 1560, in process_iter
    list(dict.fromkeys(new_pids).items()))
TypeError: '<' not supported between instances of 'NoneType' and 'Process'"   
  python_logger: "src.config.exceptions"   

I tried to include everything that would make it as close to production environment and complexity as possible
Code sample:
https://hastebin.com/akiwulohig.py
(excessive code here and there, but I hope it's readable)

@lukeboyes-serato
Copy link

lukeboyes-serato commented Apr 27, 2021

Hi there,

We also get this in our production environment ~2 times a week and also use it from multiple threads.

File "atvirtualenv3/lib/python3.7/site-packages/psutil/__init__.py", line 1434, in process_iter list(dict.fromkeys(new_pids).items())) TypeError: '<' not supported between instances of 'NoneType' and 'Process'

I wonder if moving the lock to wrap this whole code block would work?

psutil/psutil/__init__.py

Lines 1426 to 1435 in ea5b2df

a = set(pids())
b = set(_pmap.keys())
new_pids = a - b
gone_pids = b - a
for pid in gone_pids:
remove(pid)
with _lock:
ls = sorted(list(_pmap.items()) +
list(dict.fromkeys(new_pids).items()))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants