imap from ThreadPool hangs by an exception in a generator function

[lev-veshnyakov]

BPO	28696
Nosy	@applio, @zhangyangyu, @lev-veshnyakov
Superseder	bpo-28699: Imap from ThreadPool behaves unexpectedly
Files	test.py: Example of code leading to the issue

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-11-25.22:06:42.835>
created_at = <Date 2016-11-15.12:52:50.751>
labels = ['type-bug', 'library']
title = 'imap from ThreadPool hangs by an exception in a generator function'
updated_at = <Date 2016-11-25.22:06:42.833>
user = 'https://github.com/lev-veshnyakov'

bugs.python.org fields:

activity = <Date 2016-11-25.22:06:42.833>
actor = 'davin'
assignee = 'none'
closed = True
closed_date = <Date 2016-11-25.22:06:42.835>
closer = 'davin'
components = ['Library (Lib)']
creation = <Date 2016-11-15.12:52:50.751>
creator = 'lev-veshnyakov'
dependencies = []
files = ['45486']
hgrepos = []
issue_num = 28696
keywords = []
message_count = 10.0
messages = ['280833', '280838', '280840', '280842', '280863', '280866', '280867', '280868', '281032', '281743']
nosy_count = 3.0
nosy_names = ['davin', 'xiang.zhang', 'lev-veshnyakov']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '28699'
type = 'behavior'
url = 'https://bugs.python.org/issue28696'
versions = ['Python 3.4']

[lev-veshnyakov]

It's only in imap, in map it's ok. The following code explains the issue:

from multiprocessing.pool import ThreadPool

pool = ThreadPool(10)

def gen():
    yield 1 + '1' # here is an error

try:
next((pool.imap(str, gen())))
except:
# Will be catched using pool.map normally, but using pool.imap will be not.
# Instead it hangs. This is the same for ThreadPool and Pool.
print('this will not be printed because thread is hanging')

[applio]

Using the supplied example, I was unable to reproduce what you described using either 3.5 or 3.6-beta on OS X 10.11.

What platform are you using? (Perhaps it is platform specific.)

[lev-veshnyakov]

Ubuntu 14.04 LTS, 3.13.0-83-generic, x86_64

[lev-veshnyakov]

So, I've checked twice, it's presented by me on python 3.4.3, and not by 3.5.2. So I will go deaper

[lev-veshnyakov]

I've reproduced it on 2 different machines:

• on a MacBook in Docker (debian:jessie, python 3.4.2)
• on another desktop (Ubuntu 14.04.1, 3.16.0-77-generic, x86_64, python 3.4.3)

[lev-veshnyakov]

It's hanging in a while loop in _handle_workers, /usr/lib/python3.4/multiprocessingpool.py:365.
I can't figure out what is the reason.

@staticmethod
def _handle_workers(pool):
    thread = threading.current_thread()

    # Keep maintaining workers until the cache gets drained, unless the pool
    # is terminated.
    while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
        pool._maintain_pool()
        time.sleep(0.1)
    # send sentinel to stop workers
    pool._taskqueue.put(None)
    util.debug('worker handler exiting')

[applio]

If it only occurs in 3.4.x, is moving to 3.5 an option for you?

[lev-veshnyakov]

Yes, I'm free to move to 3.5, now I'm seeing isn't there any problems in 3.5 according to this issue.

[zhangyangyu]

In Py3.6, it raises error:

>>> next((pool.imap(str, gen())))
Traceback (most recent call last):
  File "/opt/lib/python3.7/multiprocessing/pool.py", line 684, in next
    item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lib/python3.7/multiprocessing/pool.py", line 690, in next
    item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/lib/python3.7/multiprocessing/pool.py", line 693, in next
    raise StopIteration
StopIteration

But this doesn't looks good. I think it should raise TypeError. It can be fixed in bpo-28699.

[applio]

Closing this issue -- all further discussion moves to bpo-28699