New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MultiIndexHashEngine can't manage mixed type tuples #18520

Closed
toobaz opened this Issue Nov 27, 2017 · 4 comments

Comments

Projects
None yet
3 participants
@toobaz
Member

toobaz commented Nov 27, 2017

Code Sample, a copy-pastable example if possible

In [2]: mi = pd.MultiIndex.from_product([[True, False], range(1, 10)])

In [3]: mi.get_loc((False, 1))
Out[3]: 9

In [4]: mi = pd.MultiIndex.from_product([[True, False], range(1, 10000)])

In [5]: mi.get_loc((False, 1))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-68f8698878ec> in <module>()
----> 1 mi.get_loc((False, 1))

/home/pietro/nobackup/repo/pandas/pandas/core/indexes/multi.py in get_loc(self, key, method)
   2134             key = _values_from_object(key)
   2135             key = tuple(map(_maybe_str_to_time_stamp, key, self.levels))
-> 2136             return self._engine.get_loc(key)
   2137 
   2138         # -- partial selection or non-unique index

/home/pietro/nobackup/repo/pandas/pandas/_libs/index.pyx in pandas._libs.index.MultiIndexHashEngine.get_loc (pandas/_libs/index.c:15854)()

/home/pietro/nobackup/repo/pandas/pandas/_libs/index.pyx in pandas._libs.index.MultiIndexHashEngine.get_loc (pandas/_libs/index.c:15701)()

/home/pietro/nobackup/repo/pandas/pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.MultiIndexHashTable.get_item (pandas/_libs/hashtable.c:24621)()

/home/pietro/nobackup/repo/pandas/pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.MultiIndexHashTable.get_item (pandas/_libs/hashtable.c:24468)()

/home/pietro/nobackup/repo/pandas/pandas/core/indexes/multi.py in _hashed_indexing_key(self, key)
    819         key = tuple([f(k, stringify)
    820                      for k, stringify in zip(key, self._have_mixed_levels)])
--> 821         return hash_tuple(key)
    822 
    823     @Appender(base._shared_docs['duplicated'] % _index_doc_kwargs)

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in hash_tuple(val, encoding, hash_key)
    186               for v in val)
    187 
--> 188     h = _combine_hash_arrays(hashes, len(val))[0]
    189 
    190     return h

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in _combine_hash_arrays(arrays, num_items)
     31     """
     32     try:
---> 33         first = next(arrays)
     34     except StopIteration:
     35         return np.array([], dtype=np.uint64)

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in <genexpr>(.0)
    184     """
    185     hashes = (_hash_scalar(v, encoding=encoding, hash_key=hash_key)
--> 186               for v in val)
    187 
    188     h = _combine_hash_arrays(hashes, len(val))[0]

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in _hash_scalar(val, encoding, hash_key)
    330 
    331     return hash_array(vals, hash_key=hash_key, encoding=encoding,
--> 332                       categorize=False)

/home/pietro/nobackup/repo/pandas/pandas/core/util/hashing.py in hash_array(vals, encoding, hash_key, categorize)
    290 
    291         try:
--> 292             vals = hashing.hash_object_array(vals, hash_key, encoding)
    293         except TypeError:
    294             # we have mixed types

/home/pietro/nobackup/repo/pandas/pandas/_libs/hashing.pyx in pandas._libs.hashing.hash_object_array (pandas/_libs/hashing.c:1764)()

ValueError: Does not understand character buffer dtype format string ('?')

In [6]: mi = pd.MultiIndex.from_product([[1, 0], range(1, 10000)])

In [7]: mi.get_loc((1, 1))
Out[7]: 0

Problem description

The two engines should give the same result.

Expected Output

Out[3]:

Output of pd.show_versions()

INSTALLED VERSIONS

commit: f745e52
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-3-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.22.0.dev0+241.gf745e52e1.dirty
pytest: 3.2.3
pip: 9.0.1
setuptools: 36.7.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.0dev
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.0
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1

@gfyoung

This comment has been minimized.

Member

gfyoung commented Nov 27, 2017

Complex dtypes are probably not pandas cup of tea. That being said, we should still try to make our hashtables more robust. Have a look around and see what you can find.

@jreback

This comment has been minimized.

Contributor

jreback commented Nov 27, 2017

@gfyoung
complex dtypes are pandas cup of tea, by-definition a tuple IS mixed dtype. so this is for sure in scope.

@gfyoung

This comment has been minimized.

Member

gfyoung commented Nov 27, 2017

complex dtypes are pandas cup of tea, by-definition a tuple IS mixed dtype

Okay, good to know. I was under a different impression as you can tell.

@toobaz

This comment has been minimized.

Member

toobaz commented Nov 27, 2017

Well, I guess that in this specific case, the fact that the tuple is "complex" or not should be irrelevant, because it is just a container for values that refer to different levels - and the levels are homogeneous and nice, and the values are of the desired dtypes. So yes, this should be pretty standard.

(As opposed, for instance, to a tuple as a key for a flat index - which, ironically, works just fine)

@toobaz toobaz referenced this issue Jan 4, 2018

Merged

REF: codes-based MultiIndex engine #19074

3 of 3 tasks complete

@jreback jreback added this to the 0.23.0 milestone Jan 17, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment