ProcessPoolExecutor().map() fails following an identical map()

[OceanEngineer]

BPO	24213
Nosy	@ned-deily
Files	processExample.py: bug creation

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-05-17.04:37:18.166>
created_at = <Date 2015-05-16.22:53:14.029>
labels = ['library', 'type-crash']
title = 'ProcessPoolExecutor().map() fails following an identical map()'
updated_at = <Date 2015-05-17.04:37:18.154>
user = 'https://bugs.python.org/OceanEngineer'

bugs.python.org fields:

activity = <Date 2015-05-17.04:37:18.154>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2015-05-17.04:37:18.166>
closer = 'ned.deily'
components = ['Library (Lib)']
creation = <Date 2015-05-16.22:53:14.029>
creator = 'OceanEngineer'
dependencies = []
files = ['39399']
hgrepos = []
issue_num = 24213
keywords = []
message_count = 2.0
messages = ['243375', '243383']
nosy_count = 2.0
nosy_names = ['ned.deily', 'OceanEngineer']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue24213'
versions = ['Python 3.4']

[OceanEngineer]

Attached file runs fine. Uncommenting line 19 makes python crash. Also, no crash happens if order of execution of map() and executor.map() is switched.

This problem seems to be related to numpy in some way, the code does not crash if the commented return of function user_square is substituted for numpy functions.

Using a mac with Yosemite 10.10.3. When program crashed I get window that says:

"
Python quit unexpectedly.

Click Reopen to open the application again. Click Report to see more detailed information and send a report to Apple.
"

This is the traceback:

Traceback (most recent call last):
  File "processExample.py", line 27, in <module>
    main()
  File "processExample.py", line 22, in main
    result3 = list(executor.map(user_square, nums))
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/concurrent/futures/_base.py", line 549, in result_iterator
    yield future.result()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/concurrent/futures/_base.py", line 402, in result
    return self.__get_result()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

[ned-deily]

I can reproduce a crash with your sample program using the current numpy wheel (1.9.2) and the python.org 3.4.3. The significant parts of the OS X crash dump:

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000110

VM Regions Near 0x110:
-->
__TEXT 0000000100000000-0000000100001000 [ 4K] r-x/rwx SM=COW /Library/Frameworks/Python.framework/Versions/3.4.3_10_6/Resources/Python.app/Contents/MacOS/Python

Application Specific Information:
crashed on child side of fork pre-exec

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libdispatch.dylib 0x00007fff97347c13 dispatch_group_async + 533
1 libBLAS.dylib 0x00007fff96271a67 dsyrk_r + 565
2 libBLAS.dylib 0x00007fff96272183 APL_dsyrk + 370
3 libBLAS.dylib 0x00007fff96370696 cblas_dgemm + 1144
4 _dotblas.so 0x0000000104216775 dotblas_matrixproduct + 5733
5 org.python.python 0x00000001000e44ce PyEval_EvalFrameEx + 28894
6 org.python.python 0x00000001000e5ced PyEval_EvalCodeEx + 2349
7 org.python.python 0x000000010003e8ba function_call + 186
8 org.python.python 0x000000010000d3c8 PyObject_Call + 104
9 org.python.python 0x00000001000e0cb9 PyEval_EvalFrameEx + 14537
10 org.python.python 0x00000001000e5ced PyEval_EvalCodeEx + 2349
11 org.python.python 0x000000010003e8ba function_call + 186
12 org.python.python 0x000000010000d3c8 PyObject_Call + 104
13 org.python.python 0x00000001000e0cb9 PyEval_EvalFrameEx + 14537
14 org.python.python 0x00000001000e4455 PyEval_EvalFrameEx + 28773
15 org.python.python 0x00000001000e4455 PyEval_EvalFrameEx + 28773
16 org.python.python 0x00000001000e4455 PyEval_EvalFrameEx + 28773
17 org.python.python 0x00000001000e5ced PyEval_EvalCodeEx + 2349
18 org.python.python 0x000000010003e8ba function_call + 186
19 org.python.python 0x000000010000d3c8 PyObject_Call + 104
20 org.python.python 0x000000010002802c method_call + 140
21 org.python.python 0x000000010000d3c8 PyObject_Call + 104
22 org.python.python 0x000000010007b831 slot_tp_init + 81
23 org.python.python 0x0000000100072d14 type_call + 212
24 org.python.python 0x000000010000d3c8 PyObject_Call + 104
25 org.python.python 0x00000001000e1b09 PyEval_EvalFrameEx + 18201
26 org.python.python 0x00000001000e4455 PyEval_EvalFrameEx + 28773
27 org.python.python 0x00000001000e4455 PyEval_EvalFrameEx + 28773
28 org.python.python 0x00000001000e4455 PyEval_EvalFrameEx + 28773
29 org.python.python 0x00000001000e4455 PyEval_EvalFrameEx + 28773
30 org.python.python 0x00000001000e4455 PyEval_EvalFrameEx + 28773
31 org.python.python 0x00000001000e5ced PyEval_EvalCodeEx + 2349
32 org.python.python 0x000000010003e8ba function_call + 186
33 org.python.python 0x000000010000d3c8 PyObject_Call + 104
34 org.python.python 0x00000001000e0cb9 PyEval_EvalFrameEx + 14537
35 org.python.python 0x00000001000e5ced PyEval_EvalCodeEx + 2349
36 org.python.python 0x00000001000e4306 PyEval_EvalFrameEx + 28438
37 org.python.python 0x00000001000e5ced PyEval_EvalCodeEx + 2349
38 org.python.python 0x00000001000e4306 PyEval_EvalFrameEx + 28438
39 org.python.python 0x00000001000e4455 PyEval_EvalFrameEx + 28773
40 org.python.python 0x00000001000e5ced PyEval_EvalCodeEx + 2349
41 org.python.python 0x00000001000e5daf PyEval_EvalCode + 63
42 org.python.python 0x000000010011048e PyRun_FileExFlags + 206
43 org.python.python 0x000000010011083d PyRun_SimpleFileExFlags + 717
44 org.python.python 0x000000010012810e Py_Main + 3262
45 org.python.python 0x0000000100000e32 0x100000000 + 3634
46 org.python.python 0x0000000100000c84 0x100000000 + 3204

Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000105001480 rbx: 0x0000000000000000 rcx: 0x0000000000000000 rdx: 0x0000000000000100
rdi: 0x00007fff7cd5fd40 rsi: 0x0000000000000000 rbp: 0x00007fff5bffbbd0 rsp: 0x00007fff5bffbba0
r8: 0x0000000105001400 r9: 0x0000000000000000 r10: 0x00000000000007d0 r11: 0xffff8001a446c4d8
r12: 0x00000001004682b0 r13: 0x0000000100437210 r14: 0x00007fff97341317 r15: 0x0000000000002000
rip: 0x00007fff97347c13 rfl: 0x0000000000010206 cr2: 0x0000000000000110

Logical CPU: 2
Error Code: 0x00000006
Trap Number: 14
[...]
0x104212000 - 0x104217ff7 +_dotblas.so (???) <15CB5D81-FD91-3005-A21C-A7D1432020C2> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/numpy/core/_dotblas.so
[...]
0x7fff96258000 - 0x7fff963e6fff libBLAS.dylib (1128) <497912C1-A98E-3281-BED7-E9C751552F61> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
[...]

So the crash is occurring as a result of a call from NumPy's _dotblas extension module to OS X's libBLAS. There's nothing obvious here that would indicate this is a problem in Python itself. Suggest you pursue this in the NumPy community and, if necessary, open an issue there:

http://www.scipy.org/scipylib/mailing-lists.html