Objmode crashes sometimes

[ehsantn]

Reporting a bug

• I am using the latest released master version of Numba (most recent is visible in
  the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
• I have included below a minimal working reproducer (if you are unsure how
  to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

A function that has objmode inside it crashes sometimes in HPAT. I haven't been able to create a small reproducer yet, but the backtrace from gdb is below. Any ideas?

0x000055555568cfd4 in PyList_GetItem ()
(gdb) bt
#0  0x000055555568cfd4 in PyList_GetItem ()
#1  0x00007ffff7e5308a in hpat::csv_ext::csv_reader_py$249(pyobject) ()
#2  0x00007ffff7e53b95 in cpython::hpat::csv_ext::csv_reader_py$249(pyobject) ()
#3  0x0000555555662b94 in _PyCFunction_FastCallDict ()
#4  0x0000555555662faf in _PyObject_FastCallDict ()
#5  0x00005555556812ed in PyObject_CallFunctionObjArgs ()
#6  0x00007ffff7e51135 in hpat::csv_ext::csv_reader_py$2410(StringType) ()
#7  0x00007ffff7e4b0b8 in __main__::TestIO::test_csv_parallel1::$3clocals$3e::test_impl$241 ()
#8  0x00007ffff7e4b5e6 in cpython::__main__::TestIO::test_csv_parallel1::$3clocals$3e::test_impl$241 ()
#9  0x00007fffd1c66584 in call_cfunc () from /home/etotoni/pse-hpc/python/numba/numba/_dispatcher.cpython-36m-x86_64-linux-gnu.so
#10 0x00007fffd1c669c9 in compile_and_invoke () from /home/etotoni/pse-hpc/python/numba/numba/_dispatcher.cpython-36m-x86_64-linux-gnu.so
#11 0x00007fffd1c66b3f in Dispatcher_call () from /home/etotoni/pse-hpc/python/numba/numba/_dispatcher.cpython-36m-x86_64-linux-gnu.so

[sklam]

I can start looking into it if you can provide the definition of the csv_reader_py.

btw, can you try moving bulk of the logic inside the objmode contextmanager into it's own python function? objmode body can just call arbitrary python function.

[ehsantn]

The function is below (execed though). For reference, this is generated for a test in a HPAT PR.

def csv_reader_py(fname):
  f_reader = csv_file_chunk_reader(fname, True)
  with objmode(A='int64[::1]', B='float64[::1]', C='float64[::1]', D='int64[::1]'):
    df = pd.read_csv(f_reader, names=['A', 'B', 'C', 'D'],
       parse_dates=[])
    A = df.A.values
    B = df.B.values
    C = df.C.values
    D = df.D.values
  return (A, B, C, D,)

[ehsantn]

Sure, I'll try using a separate function as well.

[ehsantn]

The same crash when I move the logic inside another function.

[ehsantn]

Looks like it doesn't crash if the function inside objmode has a typer available! For example, calling pd.Series() crashes but adding the typer below makes it work. Any idea?

@infer_global(pd.Series)
class DummyTyper(AbstractTemplate):
    def generic(self, args, kws):
        return signature(types.none)

[ehsantn]

This seems random; doesn't always work. Managed to get an error instead of crash: ValueError: size mismatch for tuple, expected 0 element(s) but got -1.

[ehsantn]

I think I found the issue. The lowerer hard-codes the address to the lifted function in the binary here. However, the reference to the function can be lost, resulting in memory error. Keeping an extra reference around fixes the issue for me.