Segfault on clean-up with count example from docstrings

[jorisvandenbossche]

In #19952 I have problems with a segfault when running doctests, and I could isolate at least the following code snippet that segfaults on clean-up (so only when exiting the process; an explicit del + gc.collect does not yet trigger the segfault).

import numpy as  np
import pandas as pd

df = pd.DataFrame({"Person": ["John", "Myla", None, "John", "Myla"],
                   "Age": [24., np.nan, 21., 33, 26],
                   "Single": [False, True, True, True, False]})
                  
res = df.set_index(["Person", "Single"]).count(level="Person")

[jorisvandenbossche]

OK, and the cause is the missing value in the 'Person' level.

Simplified example:

In [1]: df = pd.DataFrame({"A": ["a", "b", None, "a", "b"],
   ...:                    "B": 1,
   ...:                    "C": range(5)})
   ...:                   
   ...: res = df.set_index(["A", "B"]).count(level="A")

In [2]: res
Out[2]: 
   B
A   
a  2
b  2

is causing the segfault (and without the None it does not).
But, when selecting the column, it does not produce a segfault, but it also gives a different result (the NaN is included in index):

In [1]: df = pd.DataFrame({"A": ["a", "b", None, "a", "b"],
   ...:                    "B": 1,
   ...:                    "C": range(5)})
   ...:                   
   ...: res = df.set_index(["A", "B"])['C'].count(level="A")

In [2]: res
Out[2]: 
A
a      2
b      2
NaN    1
Name: C, dtype: int64

Here the segfault does not happen, but it seems buggy behaviour, as count should not count the NaNs ?

[jreback]

is this example fixed? is this just a tracking issue for finding the cause?

@pandas-dev/pandas-core

[gfyoung]

@jreback : Not really sure? The examples don't seem to segfault for me...

cc @jorisvandenbossche

Actually, I take that back. They do. Just as I exited the Python console.

[gfyoung]

@jorisvandenbossche : This issue has gone quiet for a little while, but I tried running your examples again, and it seems like the second example you gave, while the first and third ones are still buggy...

That's quite odd.

[gfyoung]

as count should not count the NaNs?

@jorisvandenbossche : I think that refers to values being counted. NaN can be considered one of the "groups" when using the level parameter.

[TomAugspurger]

Pushing to 0.24.2.

[jorisvandenbossche]

And directly pushed it to 0.25.0. It is not a critical bug (just a nasty one to find out)

[jorisvandenbossche]

is this just a tracking issue for finding the cause?

Yes. From what I recall, somewhere in count an invalid dataframe gets created. Or at least, it doesn't look invalid (the repr is fine), but after that when closing the python session, it segfaults. Didn't get any further than that I think.

[jreback]

is this still an issue?

[jorisvandenbossche]

Re-ran the example in the second post, and yes, this still segfaults on clean-up

[mroeschke]

Both examples are working on master. Suppose this could use a regression test.

In [28]: In [1]: df = pd.DataFrame({"A": ["a", "b", None, "a", "b"],
    ...:    ...:                    "B": 1,
    ...:    ...:                    "C": range(5)})
    ...:    ...:
    ...:    ...: res = df.set_index(["A", "B"]).count(level="A")

In [29]: res
Out[29]:
   C
A
a  2
b  2

In [30]: In [1]: df = pd.DataFrame({"A": ["a", "b", None, "a", "b"],
    ...:    ...:                    "B": 1,
    ...:    ...:                    "C": range(5)})
    ...:    ...:
    ...:    ...: res = df.set_index(["A", "B"])['C'].count(level="A")

In [31]: res
Out[31]:
A
a      2
b      2
NaN    1
Name: C, dtype: int64

In [32]: pd.__version__
Out[32]: '0.26.0.dev0+555.gf7d162b18'

[jorisvandenbossche]

I still see the issue (it's only on clean-up, you have to close your session to see it)

[mroeschke]

Ah! I see. Sorry didn't realize it was on cleanup.

[WillAyd]

I couldn't get a segfault on any of these on master during execution or while exiting - still an issue?

[jorisvandenbossche]

On latest master, it now even directly segfaults for me (not only on exit):

In [1]: In [1]: df = pd.DataFrame({"A": ["a", "b", None, "a", "b"], 
   ...:    ...:                    "B": 1, 
   ...:    ...:                    "C": range(5)}) 
   ...:    ...:                    
   ...:    ...: res = df.set_index(["A", "B"]).count(level="A")                                                                                                                                                    

In [2]: corrupted size vs. prev_size                                                                                                                                                                               
Aborted (core dumped)

[WillAyd]

I was able to generate a core dump for this and as far as I can tell the segfault happens in internals.pyx and has something to do with _as_array.

Here is the generated code for internals where I think this segfault starts:

frame #8: 0x000000011eaae0a2 internals.cpython-37m-darwin.so`__pyx_tp_dealloc_6pandas_5_libs_9internals_BlockPlacement(o=0x000000011f326ad0) at internals.c:25132:3 [opt]
   25129	  #endif
   25130	  PyObject_GC_UnTrack(o);
   25131	  Py_CLEAR(p->_as_slice);
-> 25132	  Py_CLEAR(p->_as_array);
   25133	  (*Py_TYPE(o)->tp_free)(o);
   25134	}

Everything else I see in the traceback would be either numpy or stdlib

    frame #0: 0x00007fff644a37fa libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff64560bc1 libsystem_pthread.dylib`pthread_kill + 432
    frame #2: 0x00007fff6442aa1c libsystem_c.dylib`abort + 120
    frame #3: 0x00007fff64520647 libsystem_malloc.dylib`malloc_vreport + 548
    frame #4: 0x00007fff6452340e libsystem_malloc.dylib`malloc_report + 151
    frame #5: 0x000000011095eea0 _multiarray_umath.cpython-37m-darwin.so`_buffer_clear_info + 144
    frame #6: 0x000000011095ede8 _multiarray_umath.cpython-37m-darwin.so`_dealloc_cached_buffer_info + 88
    frame #7: 0x0000000110918b62 _multiarray_umath.cpython-37m-darwin.so`array_dealloc + 18
    frame #8: 0x000000011eaae0a2 internals.cpython-37m-darwin.so`__pyx_tp_dealloc_6pandas_5_libs_9internals_BlockPlacement(o=0x000000011f326ad0) at internals.c:25132:3 [opt]
    frame #9: 0x000000010e278217 python`subtype_dealloc + 1207
    frame #10: 0x000000010e274c7b python`tupledealloc + 155
  * frame #11: 0x000000010e278217 python`subtype_dealloc + 1207
    frame #12: 0x000000010e24a8f1 python`dict_dealloc + 193
    frame #13: 0x000000010e2783a6 python`subtype_dealloc + 1606
    frame #14: 0x000000010e321cbf python`call_function + 815
    frame #15: 0x000000010e31d94f python`_PyEval_EvalFrameDefault + 32863
    frame #16: 0x000000010e3152c5 python`_PyEval_EvalCodeWithName + 3317
    frame #17: 0x000000010e310bcf python`builtin_exec + 767
    frame #18: 0x000000010e218226 python`_PyMethodDef_RawFastCallKeywords + 678
    frame #19: 0x000000010e21831a python`_PyCFunction_FastCallKeywords + 42
    frame #20: 0x000000010e321bf9 python`call_function + 617
    frame #21: 0x000000010e31d89f python`_PyEval_EvalFrameDefault + 32687
    frame #22: 0x000000010e22a748 python`gen_send_ex + 280
    frame #23: 0x000000010e31899f python`_PyEval_EvalFrameDefault + 12463
    frame #24: 0x000000010e22a748 python`gen_send_ex + 280
    frame #25: 0x000000010e31899f python`_PyEval_EvalFrameDefault + 12463
    frame #26: 0x000000010e22a748 python`gen_send_ex + 280
    frame #27: 0x000000010e218294 python`_PyMethodDef_RawFastCallKeywords + 788
    frame #28: 0x000000010e222828 python`_PyMethodDescr_FastCallKeywords + 88
    frame #29: 0x000000010e321c92 python`call_function + 770
    frame #30: 0x000000010e31d7e6 python`_PyEval_EvalFrameDefault + 32502
    frame #31: 0x000000010e2179f9 python`function_code_fastcall + 249
    frame #32: 0x000000010e321c6c python`call_function + 732
    frame #33: 0x000000010e31d89f python`_PyEval_EvalFrameDefault + 32687
    frame #34: 0x000000010e2179f9 python`function_code_fastcall + 249
    frame #35: 0x000000010e321c6c python`call_function + 732
    frame #36: 0x000000010e31d7e6 python`_PyEval_EvalFrameDefault + 32502
    frame #37: 0x000000010e3152c5 python`_PyEval_EvalCodeWithName + 3317
    frame #38: 0x000000010e217f63 python`_PyFunction_FastCallKeywords + 227
    frame #39: 0x000000010e321c6c python`call_function + 732
    frame #40: 0x000000010e31d94f python`_PyEval_EvalFrameDefault + 32863
    frame #41: 0x000000010e3152c5 python`_PyEval_EvalCodeWithName + 3317
    frame #42: 0x000000010e217f63 python`_PyFunction_FastCallKeywords + 227
    frame #43: 0x000000010e321c6c python`call_function + 732
    frame #44: 0x000000010e31d7e6 python`_PyEval_EvalFrameDefault + 32502
    frame #45: 0x000000010e3152c5 python`_PyEval_EvalCodeWithName + 3317
    frame #46: 0x000000010e217f63 python`_PyFunction_FastCallKeywords + 227
    frame #47: 0x000000010e321c6c python`call_function + 732
    frame #48: 0x000000010e31d7e6 python`_PyEval_EvalFrameDefault + 32502
    frame #49: 0x000000010e2179f9 python`function_code_fastcall + 249
    frame #50: 0x000000010e321c6c python`call_function + 732
    frame #51: 0x000000010e31d7e6 python`_PyEval_EvalFrameDefault + 32502
    frame #52: 0x000000010e3152c5 python`_PyEval_EvalCodeWithName + 3317
    frame #53: 0x000000010e217161 python`_PyFunction_FastCallDict + 945
    frame #54: 0x000000010e218ae0 python`_PyObject_Call_Prepend + 144
    frame #55: 0x000000010e218511 python`PyObject_Call + 241
    frame #56: 0x000000010e31db25 python`_PyEval_EvalFrameDefault + 33333
    frame #57: 0x000000010e3152c5 python`_PyEval_EvalCodeWithName + 3317
    frame #58: 0x000000010e217f63 python`_PyFunction_FastCallKeywords + 227
    frame #59: 0x000000010e321c6c python`call_function + 732
    frame #60: 0x000000010e31d89f python`_PyEval_EvalFrameDefault + 32687
    frame #61: 0x000000010e3152c5 python`_PyEval_EvalCodeWithName + 3317
    frame #62: 0x000000010e367854 python`PyRun_FileExFlags + 244
    frame #63: 0x000000010e366dda python`PyRun_SimpleFileExFlags + 474
    frame #64: 0x000000010e39248d python`pymain_main + 11421
    frame #65: 0x000000010e1f05ed python`main + 125
    frame #66: 0x00007fff6435c7fd libdyld.dylib`start + 1