New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strange behaviour with MultiIndex and more than one million records #18165

Closed
olevold opened this Issue Nov 8, 2017 · 8 comments

Comments

Projects
None yet
2 participants
@olevold

olevold commented Nov 8, 2017

Code Sample

r = list(range(999999))
df2 = pd.DataFrame({'a':r, 'b':r}, index=pd.MultiIndex.from_tuples([(x,x) for x in r]))
df2['a'].blabla()
# result as expected: AttributeError: 'Series' object has no attribute 'blabla'

r = list(range(1000000))
df2 = pd.DataFrame({'a':r, 'b':r}, index=pd.MultiIndex.from_tuples([(x,x) for x in r]))
df2['a'].blabla()
#strange result: TypeError: unorderable types: numpy.ndarray() < str()

from scipy.stats.mstats import winsorize
winsorize(df2['a'])
#result: TypeError: unorderable types: numpy.ndarray() < str()

Problem description

When there are exactly one million records or more with a MultiIndex, I get this TypeError when I call a non-existent method on a column. I get the same error when I try to call the winsorize function from scipy.stats on a column. The type error originates here: File "pandas/index.pyx", line 481, in pandas.index._bin_search

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-98-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: nb_NO.UTF-8
LOCALE: nb_NO.UTF-8

pandas: 0.21.0
pytest: 3.2.2
pip: 8.1.1
setuptools: 20.7.0
Cython: 0.26.1
numpy: 1.13.1
scipy: 0.19.1
pyarrow: None
xarray: 0.9.6
IPython: 2.4.1
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: 2.4.4
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 8, 2017

Contributor

show versions as the issue template indicates

Contributor

jreback commented Nov 8, 2017

show versions as the issue template indicates

@olevold

This comment has been minimized.

Show comment
Hide comment
@olevold

olevold commented Nov 8, 2017

Done

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 8, 2017

Contributor

so indexing with a million elements is different internally
so we are prob not catching and reraising the correct exception here
this hits indexing because you are accessing an attribute , which u then try to call
attribute access can be for columns as well so that’s why indexing is hit

if you would trace would be great - this will end up in cython in index.pyx

Contributor

jreback commented Nov 8, 2017

so indexing with a million elements is different internally
so we are prob not catching and reraising the correct exception here
this hits indexing because you are accessing an attribute , which u then try to call
attribute access can be for columns as well so that’s why indexing is hit

if you would trace would be great - this will end up in cython in index.pyx

@jreback jreback added this to the Next Major Release milestone Nov 8, 2017

@olevold

This comment has been minimized.

Show comment
Hide comment
@olevold

olevold Nov 8, 2017

File "/usr/local/lib/python3.5/site-packages/scipy/stats/mstats_basic.py", line 1419, in winsorize
a = ma.array(a, copy=np.logical_not(inplace))
File "/usr/local/lib/python3.5/site-packages/numpy/ma/core.py", line 6152, in array
ndmin=ndmin, shrink=shrink, order=order)
File "/usr/local/lib/python3.5/site-packages/numpy/ma/core.py", line 2762, in new
_baseclass = getattr(data, '_baseclass', type(_data))
File "/usr/local/lib/python3.5/site-packages/pandas/core/generic.py", line 2742, in getattr
if name in self._info_axis:
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1042, in contains
self.get_loc(key)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1643, in get_loc
loc = self._get_level_indexer(key, level=0)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1905, in _get_level_indexer
loc = level_index.get_loc(key)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/numeric.py", line 341, in get_loc
tolerance=tolerance)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/base.py", line 2134, in get_loc
return self._engine.get_loc(key)
File "pandas/index.pyx", line 132, in pandas.index.IndexEngine.get_loc (pandas/index.c:4433)
File "pandas/index.pyx", line 140, in pandas.index.IndexEngine.get_loc (pandas/index.c:4075)
File "pandas/index.pyx", line 481, in pandas.index._bin_search (pandas/index.c:9823)
TypeError: unorderable types: numpy.ndarray() < str()

olevold commented Nov 8, 2017

File "/usr/local/lib/python3.5/site-packages/scipy/stats/mstats_basic.py", line 1419, in winsorize
a = ma.array(a, copy=np.logical_not(inplace))
File "/usr/local/lib/python3.5/site-packages/numpy/ma/core.py", line 6152, in array
ndmin=ndmin, shrink=shrink, order=order)
File "/usr/local/lib/python3.5/site-packages/numpy/ma/core.py", line 2762, in new
_baseclass = getattr(data, '_baseclass', type(_data))
File "/usr/local/lib/python3.5/site-packages/pandas/core/generic.py", line 2742, in getattr
if name in self._info_axis:
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1042, in contains
self.get_loc(key)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1643, in get_loc
loc = self._get_level_indexer(key, level=0)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1905, in _get_level_indexer
loc = level_index.get_loc(key)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/numeric.py", line 341, in get_loc
tolerance=tolerance)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/base.py", line 2134, in get_loc
return self._engine.get_loc(key)
File "pandas/index.pyx", line 132, in pandas.index.IndexEngine.get_loc (pandas/index.c:4433)
File "pandas/index.pyx", line 140, in pandas.index.IndexEngine.get_loc (pandas/index.c:4075)
File "pandas/index.pyx", line 481, in pandas.index._bin_search (pandas/index.c:9823)
TypeError: unorderable types: numpy.ndarray() < str()

@olevold

This comment has been minimized.

Show comment
Hide comment
@olevold

olevold Nov 8, 2017

BTW, the winsorize function works with 999999 records

olevold commented Nov 8, 2017

BTW, the winsorize function works with 999999 records

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 8, 2017

Contributor

ok the dundee contains methods in indexes/multi

a) move way up in the file it should be much closer to the top near other finder methods. (i know get/set state are there as well but leave those)
b) catch TypeError as well as LookupError to return False
c) test all of the cases u gave
can u do a PR?

Contributor

jreback commented Nov 8, 2017

ok the dundee contains methods in indexes/multi

a) move way up in the file it should be much closer to the top near other finder methods. (i know get/set state are there as well but leave those)
b) catch TypeError as well as LookupError to return False
c) test all of the cases u gave
can u do a PR?

@olevold

This comment has been minimized.

Show comment
Hide comment
@olevold

olevold Nov 8, 2017

I'm afraid I don't have the time to look further into it right now, I'm in the process of working around it in my project.

olevold commented Nov 8, 2017

I'm afraid I don't have the time to look further into it right now, I'm in the process of working around it in my project.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 8, 2017

Contributor

@olevold ok ; things get fixed by folks contributing

Contributor

jreback commented Nov 8, 2017

@olevold ok ; things get fixed by folks contributing

@jreback jreback modified the milestones: Next Major Release, 0.22.0, 0.21.1 Nov 12, 2017

jreback added a commit that referenced this issue Nov 12, 2017

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Dec 8, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment