Incompatibility with multiprocessing.pool in Python 2.7.x

[julkiewicz]

I get the following error when using eventlet together with gunicorn. This happens upon attempt to gracefully terminate a worker process:

File "/usr/lib/python2.7/threading.py", line 808, in __bootstrap_inner
  self.run()
File "/usr/lib/python2.7/threading.py", line 761, in run
  self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 324, in _handle_workers
  while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'

Indeed, multiprocessing.pool seems to be based on a non-public property _state which seems to have three possible states RUN CLOSE and TERMINATE. Not sure what is the difference between CLOSE and TERMINATE. The appropriate fix would probably be to implement _state as an assignable property somehow.

[smtakeda]

I came across the same error in Python 3.4.3 on Centos 6.5

Here is a test case:

from multiprocessing.pool import ThreadPool
import time
import eventlet
eventlet.monkey_patch()

def run_this(idx):
    print('hello {0}-1'.format(idx))
    time.sleep(1)
    print('hello {0}-2'.format(idx))

pool = ThreadPool(10)

for idx in range(10):
    pool.apply_async(run_this, [idx])

pool.close()
pool.join()

(eventlet_requests)stakeda@stakeda /tmp>python test.py 
Exception in thread Thread-11:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 365, in _handle_workers
    while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'

Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 379, in _handle_tasks
    if thread._state:
AttributeError: '_MainThread' object has no attribute '_state'

(eventlet_requests) $>pip list
eventlet (0.18.1)
greenlet (0.4.9)
pip (7.1.2)
setuptools (18.2)
wheel (0.24.0)

[booxter]

When we monkey patch, we get _MainThread from current_thread call. But when we don't monkey patch, we get a proper Thread that got its _state attribute assigned inside init of the ThreadPool. In this init, when self._worker_handler is initialized, it gets back a proper Thread object on which we set the _state attribute to RUN, but when we then call current_thread, we don't get the object but _MainThread, so the state is not there.

[karthikvadla]

@booxter @julkiewicz You guys found any fix for above. ?
I'm using

• python (2.7.13)
• eventlet (0.18.2)
• gunicorn (19.7.1)
• gevent (1.2.2)

I have celery task which makes call to some other non-celery task.
Then throws below.

[2017-12-09 20:34:33,404: WARNING/MainProcess] Exception in thread Thread-94:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 325, in _handle_workers
    while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'
[2017-12-09 20:34:33,498: WARNING/MainProcess] Exception in thread Thread-113:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 325, in _handle_workers
    while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'

[LingxianKong]

I met with the same problem in openstack, when using kubernetes python client(which is using multiprocessing.pool to send api request) in an oslo service(which is based on eventlet)

[hoangphuocbk]

I also met the same error, when I use kubernetes python client in OpenStack, it raise problem in multiprocessing.