BUG: segfault with .loc indexing #5553

TomAugspurger · 2013-11-20T03:50:01Z

I found that segfault that messed up my HDFStore earlier.

You'll need to csv files:

https://www.dropbox.com/s/55gcpkp3b8byfka/df1.csv
https://www.dropbox.com/s/mqqhfgocdg4qggp/df2.csv

To reproduce:

In [1]: df1 = pd.read_csv('df1.csv', index_col=[0,1,2])

In [2]: df2 = pd.read_csv('df2.csv', index_col=[0,1,2])

In [3]: df2.loc[df1.index]
Segmentation fault: 11
(pandas-dev)dhcp80fff527:Desktop tom$

I've never debugged a segfault before, but I gather there are some methods. Can anyone reproduce this?

Also, that's a valid use of .loc right?

In [1]: pd.version
Out[1]: '0.12.0-1084-ged64416'

In [2]: np.version
Out[2]: '1.7.1'

The text was updated successfully, but these errors were encountered:

TomAugspurger · 2013-11-20T03:58:23Z

Interesting... maybe...

(gdb) bt
#0  0x0000000103408e07 in __pyx_pw_6pandas_5algos_333take_2d_axis1_float64_float64
(__pyx_self=0x1, __pyx_args=<value temporarily unavailable, due to optimizations>,
__pyx_kwds=<value temporarily unavailable, due to optimizations>) at pandas/algos.c:81901

...

TomAugspurger · 2013-11-20T04:12:44Z

A potential hint: my df2.index is not unique. I'm doing the same operation on a bunch of files; for this one I have to mess with the index (all of the other indices are unique and none of those have caused any problems).

TomAugspurger · 2013-11-20T04:32:43Z

Dropping the duplicates fixed the problem:

In [34]: dupes = df2.index.get_duplicates()

In [35]: df3 = df2.drop(dupes)

In [36]: df3.loc[df1.index]
Out[36]: (not a segfault)

jreback · 2013-11-20T04:33:01Z

if one of these is not unique this prob won't work

could do this via inner join I think

TomAugspurger · 2013-11-20T04:51:12Z

I probably should have been doing that originally.

Would it be a big perf hit to check for uniqueness here?

On Nov 19, 2013, at 22:33, "jreback" <notifications@github.com mailto:notifications@github.com> wrote:

if one of these is not unique this prob won't work

could do this via inner join I think

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5553#issuecomment-28863157.

jreback · 2013-11-20T04:56:29Z

unique check is cheap
loc dispatches based on it in fact

non unique loc returns duplicate matches in order where they r found
it's possible the mi blow it up if not sorted ( just a guess why it's faulting )
can u step thru and see where fault happens - use pdb before the loc and step thru

TomAugspurger · 2013-11-20T05:12:47Z

> /Users/tom/Desktop/seg.py(6)<module>()
      5 import ipdb; ipdb.set_trace()
----> 6 df2.loc[df1.index]
      7 

ipdb> s
--Call--
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/generic.py(924)_indexer()
    923 
--> 924         def _indexer(self):
    925             if getattr(self, iname, None) is None:

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/generic.py(925)_indexer()
    924         def _indexer(self):
--> 925             if getattr(self, iname, None) is None:
    926                 setattr(self, iname, indexer(self, name))

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/generic.py(926)_indexer()
    925             if getattr(self, iname, None) is None:
--> 926                 setattr(self, iname, indexer(self, name))
    927             return getattr(self, iname)

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/generic.py(927)_indexer()
    926                 setattr(self, iname, indexer(self, name))
--> 927             return getattr(self, iname)
    928 

ipdb> n
--Return--
<pandas....04850fd0>
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/generic.py(927)_indexer()
    926                 setattr(self, iname, indexer(self, name))
--> 927             return getattr(self, iname)
    928 

ipdb> n
--Call--
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(961)__getitem__()
    960 
--> 961     def __getitem__(self, key):
    962         if type(key) is tuple:

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(962)__getitem__()
    961     def __getitem__(self, key):
--> 962         if type(key) is tuple:
    963             return self._getitem_tuple(key)

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(965)__getitem__()
    964         else:
--> 965             return self._getitem_axis(key, axis=0)
    966 

fish: Job 1, 'python seg.py' terminated by signal SIGSEGV (Address boundary error)

Stepping into that last one:

ipdb> l 965
    960 
    961     def __getitem__(self, key):
    962         if type(key) is tuple:
    963             return self._getitem_tuple(key)
    964         else:
--> 965             return self._getitem_axis(key, axis=0)
    966 
    967     def _getitem_axis(self, key, axis=0):
    968         raise NotImplementedError()
    969 
    970     def _getbool_axis(self, key, axis=0):

ipdb> s
--Call--
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(1058)_getitem_axis()
   1057 
-> 1058     def _getitem_axis(self, key, axis=0):
   1059         labels = self.obj._get_axis(axis)

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(1059)_getitem_axis()
   1058     def _getitem_axis(self, key, axis=0):
-> 1059         labels = self.obj._get_axis(axis)
   1060 

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(1061)_getitem_axis()
   1060 
-> 1061         if isinstance(key, slice):
   1062             self._has_valid_type(key,axis)

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(1064)_getitem_axis()
   1063             return self._get_slice_axis(key, axis=axis)
-> 1064         elif com._is_bool_indexer(key):
   1065             return self._getbool_axis(key, axis=axis)

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(1066)_getitem_axis()
   1065             return self._getbool_axis(key, axis=axis)
-> 1066         elif _is_list_like(key) and not (isinstance(key, tuple) and
   1067                                          isinstance(labels, MultiIndex)):

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(1069)_getitem_axis()
   1068 
-> 1069             if hasattr(key, 'ndim') and key.ndim > 1:
   1070                 raise ValueError('Cannot index with multidimensional key')

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(1072)_getitem_axis()
   1071 
-> 1072             return self._getitem_iterable(key, axis=axis)
   1073         else:

ipdb> n
fish: Job 1, 'python seg.py' terminated by signal SIGSEGV (Address boundary error)

A ways further along:

ipdb> n
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(806)_getitem_iterable()
    805 
--> 806                     result = result._reindex_with_indexers({ axis : [ new_labels, new_indexer ] }, copy=True, allow_dups=True)
    807 

ipdb> n
fish: Job 1, 'python seg.py' terminated by signal SIGSEGV (Address boundary error)

TomAugspurger · 2013-11-20T05:19:38Z

a bit further:

> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.py(806)_getitem_iterable()
    805 
--> 806                     result = result._reindex_with_indexers({ axis : [ new_labels, new_indexer ] }, copy=True, allow_dups=True)
    807 

ipdb> s
--Call--
> /Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.12.0_1084_ged64416-py2.7-macosx-10.8-x86_64.egg/pandas/core/generic.py(1400)_reindex_with_indexers()
   1399 
-> 1400     def _reindex_with_indexers(self, reindexers, method=None, fill_value=np.nan, limit=None, copy=False, allow_dups=False):
   1401         """ allow_dups indicates an internal call here """

ipdb> args
self = <class 'pandas.core.frame.DataFrame'>
MultiIndex: 13343 entries, (61002006171, 1, 1.0) to (999990398540913, 1, 2.0)
Data columns (total 71 columns):
PUHROFF2     504  non-null values
PWSSWGT      13343  non-null values
PUHROFF1     5838  non-null values
PEERNHRO     0  non-null values
PEERNHRY     0  non-null values
PENLFRET     183  non-null values
PROLDRRP     13343  non-null values
HRLONGLK     13343  non-null values
PTDTRACE     13343  non-null values
PRFAMREL     13343  non-null values
PUWK         10729  non-null values
PUERN2       0  non-null values
PUIODP1      5978  non-null values
PUIODP2      5769  non-null values
PUIODP3      5713  non-null values
PWCMPWGT     13343  non-null values
PEMLR        10766  non-null values
PESEX        13343  non-null values
PREMPNOT     10766  non-null values
PRHRUSL      6638  non-null values
PEHRFTPT     601  non-null values
PRDTIND1     6889  non-null values
PUDIS2       19  non-null values
PUDIS1       70  non-null values
PRTAGE       13343  non-null values
PWORWGT      13343  non-null values
PEHRWANT     1195  non-null values
PEERNLAB     0  non-null values
PEEDUCA      10800  non-null values
PEHRACT2     380  non-null values
GEREG        13343  non-null values
PUERNH1C     0  non-null values
HWHHWGT      13343  non-null values
GTCBSA       13343  non-null values
PUCHINHH     13343  non-null values
HRMONTH      13343  non-null values
HRMIS        13343  non-null values
GESTFIPS     13343  non-null values
HEFAMINC     13343  non-null values
PUDIS        420  non-null values
PUHROT1      5838  non-null values
PEHRACT1     6450  non-null values
PEHRUSL2     383  non-null values
PEHRUSL1     6638  non-null values
HRINTSTA     13343  non-null values
PUABSOT      1962  non-null values
PEERN        0  non-null values
GTINDVPC     13343  non-null values
PUHROT2      492  non-null values
PULAY        1653  non-null values
PEERNPER     0  non-null values
HUFINAL      13343  non-null values
PTHR         13343  non-null values
PRDISC       97  non-null values
PTWK         13343  non-null values
PWFMWGT      13343  non-null values
GESTCEN      13343  non-null values
PERET1       1935  non-null values
PURETOT      1850  non-null values
PEMJOT       6638  non-null values
PWLGWGT      13343  non-null values
HRYEAR4      13343  non-null values
PEHRUSLT     6638  non-null values
PEMARITL     10800  non-null values
PRPERTYP     13343  non-null values
PEHRACTT     6450  non-null values
PRERNWA      0  non-null values
PTOT         13343  non-null values
PEMJNUM      383  non-null values
PELKDUR      251  non-null values
timestamp    13343  non-null values
dtypes: float64(56), int64(14), object(1)
reindexers = {0: [array([(61002006171, 1, 1.0), (61002006171, 1, 2.0), (61002006171, 1, 3.0),
       ..., None, (999990398540913, 1, 1.0), (999990398540913, 1, 2.0)], dtype=object), array([    0,     1,     2, ...,    -1, 19716, 19717])]}
method = None
fill_value = nan
limit = None
copy = True
allow_dups = True

I'll take a closer look tomorrow. It's gotten down to pandas/core/internals.py(3015)reindex_indexer()

jreback · 2013-11-20T13:14:14Z

@TomAugspurger can you post the file links again? I think they are the same

TomAugspurger · 2013-11-20T13:56:42Z

Sorry about that. Here's the other one. I'll edit the original post.

https://www.dropbox.com/s/mqqhfgocdg4qggp/df2.csv

jreback · 2013-11-20T15:28:13Z

fixed in #5555, didn't have the correct test case (which basically is selecting a unique index from a non-unique index, where you have lots of not found elements)
object? -> Details about 'object', use 'object??' for extra details.

In [1]:  df = DataFrame({'test': [5,7,9,11], 'test1': [4.,5,6,7], 'other': list('abcd') }, index=['A', 'A', 'B', 'C'])

In [2]: rows = ['F','G','H','C','B','E']

In [3]: df.loc[rows]
Out[3]: 
  other  test  test1
F   NaN   NaN    NaN
G   NaN   NaN    NaN
H   NaN   NaN    NaN
C     d    11      7
B     c     9      6
E   NaN   NaN    NaN

jreback mentioned this issue Nov 20, 2013

BUG: Bug in selecting from a non-unique index with loc (GH5553) #5555

Merged

jreback closed this as completed in #5555 Nov 20, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: segfault with .loc indexing #5553

BUG: segfault with .loc indexing #5553

TomAugspurger commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

jreback commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

jreback commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

jreback commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

jreback commented Nov 20, 2013

BUG: segfault with .loc indexing #5553

BUG: segfault with .loc indexing #5553

Comments

TomAugspurger commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

jreback commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

jreback commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

jreback commented Nov 20, 2013

TomAugspurger commented Nov 20, 2013

jreback commented Nov 20, 2013