Py3.6 threading/reference counting issues with `numexpr`

[RobertMcLeod]

BPO	31115
Nosy	@mdickinson, @pitrou

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-21.20:52:54.988>
created_at = <Date 2017-08-03.21:04:49.095>
labels = ['extension-modules', 'library', 'type-crash']
title = 'Py3.6 threading/reference counting issues with `numexpr`'
updated_at = <Date 2018-11-21.20:52:54.986>
user = 'https://bugs.python.org/RobertMcLeod'

bugs.python.org fields:

activity = <Date 2018-11-21.20:52:54.986>
actor = 'mark.dickinson'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-21.20:52:54.988>
closer = 'mark.dickinson'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2017-08-03.21:04:49.095>
creator = 'Robert McLeod'
dependencies = []
files = []
hgrepos = []
issue_num = 31115
keywords = []
message_count = 4.0
messages = ['299722', '299762', '299778', '330215']
nosy_count = 3.0
nosy_names = ['mark.dickinson', 'pitrou', 'Robert McLeod']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue31115'
versions = ['Python 3.6']

[RobertMcLeod]

I'm working on the development branch of the numexpr module and I've run into some problems on Python 3.6, where I seem to get a variety of errors relating to threading or reference counting errors. This module is commonly used for accelerating NumPy code, as it can parse an expression into a C-extension virtual machine program and break the calculations down into blocks which are then dispatched over multiple threads. I get similar errors on both Ubuntu 16.04 with GCC5.4 and Win7 with MSVC2015. I created an issue here where I provide some of the gcc error outputs.

pydata/numexpr#252

Typically I'm getting a different error on every run.

We use a wrapper for Windows threads that emulated pthreads, and I found on Windows crashes occurred on calling the Windows system function WaitForSingleObject(cond->sema, INFINITE); in the file numexpr3/win32/pthread.c.

I cannot replicate this problem in Python 2.7/3.4/3.5 on Windows or Linux. I'm using Anaconda in both instances, with nomkl NumPy on Linux and mkl NumPy on Windows.

I tried valgrinding with Python 3.5 and 3.6, and I get numerous errors coming from places like pickle and ast (which the new NumExpr uses) in 3.6 and it's basically clean in 3.5. The logs are attached to the issue linked above.

[RobertMcLeod]

After building with Python3.7 I was able to get a useful error message that PyMem functions were being called inside GIL release. I will replace these with C-equivalents and try with Python 3.6.

Fatal Python error: Python memory allocator called without holding the GIL

Thread 0x0000000004044e00 (most recent call first):
  File "/home/rmcleod/py37/lib/python3.7/site-packages/numexpr3-3.0==19176== 
==19176== Process terminating with default action of signal 6 (SIGABRT)
==19176==    at 0x579C428: raise (raise.c:54)
==19176==    by 0x579E029: abort (abort.c:89)
==19176==    by 0x422DF7: Py_FatalError (pylifecycle.c:1849)
==19176==    by 0x41ED4C: _PyMem_DebugCheckGIL (obmalloc.c:1972)
==19176==    by 0x41ED23: _PyMem_DebugMalloc (obmalloc.c:1980)
==19176==    by 0x41FCAC: PyMem_Malloc (obmalloc.c:418)
==19176==    by 0xCEA1920: NumExprObject_copy_threadsafe(NumExprObject const*) (interpreter.cpp:147)
==19176==    by 0xCEA77CE: th_worker(void*) (module.cpp:73)
==19176==    by 0x4E416B9: start_thread (pthread_create.c:333)

[pitrou]

If, judging by pydata/numexpr@07d9245, you are really calling a bunch of C API functions without holding the GIL, then it's not surprising you may get crashes all over the place.

[mdickinson]

Judging by the current state of pydata/numexpr#252, it does seem as though this is resolved downstream, so probably safe to close here. Robert: feel free to reopen if I misunderstood.