New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MultiIndexHashEngine can't manage mixed type tuples #18520

toobaz opened this Issue Nov 27, 2017 · 4 comments


None yet
3 participants

toobaz commented Nov 27, 2017

Code Sample, a copy-pastable example if possible

In [2]: mi = pd.MultiIndex.from_product([[True, False], range(1, 10)])

In [3]: mi.get_loc((False, 1))
Out[3]: 9

In [4]: mi = pd.MultiIndex.from_product([[True, False], range(1, 10000)])

In [5]: mi.get_loc((False, 1))
ValueError                                Traceback (most recent call last)
<ipython-input-5-68f8698878ec> in <module>()
----> 1 mi.get_loc((False, 1))

/home/pietro/nobackup/repo/pandas/pandas/core/indexes/ in get_loc(self, key, method)
   2134             key = _values_from_object(key)
   2135             key = tuple(map(_maybe_str_to_time_stamp, key, self.levels))
-> 2136             return self._engine.get_loc(key)
   2138         # -- partial selection or non-unique index

/home/pietro/nobackup/repo/pandas/pandas/_libs/index.pyx in pandas._libs.index.MultiIndexHashEngine.get_loc (pandas/_libs/index.c:15854)()

/home/pietro/nobackup/repo/pandas/pandas/_libs/index.pyx in pandas._libs.index.MultiIndexHashEngine.get_loc (pandas/_libs/index.c:15701)()

/home/pietro/nobackup/repo/pandas/pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.MultiIndexHashTable.get_item (pandas/_libs/hashtable.c:24621)()

/home/pietro/nobackup/repo/pandas/pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.MultiIndexHashTable.get_item (pandas/_libs/hashtable.c:24468)()

/home/pietro/nobackup/repo/pandas/pandas/core/indexes/ in _hashed_indexing_key(self, key)
    819         key = tuple([f(k, stringify)
    820                      for k, stringify in zip(key, self._have_mixed_levels)])
--> 821         return hash_tuple(key)
    823     @Appender(base._shared_docs['duplicated'] % _index_doc_kwargs)

/home/pietro/nobackup/repo/pandas/pandas/core/util/ in hash_tuple(val, encoding, hash_key)
    186               for v in val)
--> 188     h = _combine_hash_arrays(hashes, len(val))[0]
    190     return h

/home/pietro/nobackup/repo/pandas/pandas/core/util/ in _combine_hash_arrays(arrays, num_items)
     31     """
     32     try:
---> 33         first = next(arrays)
     34     except StopIteration:
     35         return np.array([], dtype=np.uint64)

/home/pietro/nobackup/repo/pandas/pandas/core/util/ in <genexpr>(.0)
    184     """
    185     hashes = (_hash_scalar(v, encoding=encoding, hash_key=hash_key)
--> 186               for v in val)
    188     h = _combine_hash_arrays(hashes, len(val))[0]

/home/pietro/nobackup/repo/pandas/pandas/core/util/ in _hash_scalar(val, encoding, hash_key)
    331     return hash_array(vals, hash_key=hash_key, encoding=encoding,
--> 332                       categorize=False)

/home/pietro/nobackup/repo/pandas/pandas/core/util/ in hash_array(vals, encoding, hash_key, categorize)
    291         try:
--> 292             vals = hashing.hash_object_array(vals, hash_key, encoding)
    293         except TypeError:
    294             # we have mixed types

/home/pietro/nobackup/repo/pandas/pandas/_libs/hashing.pyx in pandas._libs.hashing.hash_object_array (pandas/_libs/hashing.c:1764)()

ValueError: Does not understand character buffer dtype format string ('?')

In [6]: mi = pd.MultiIndex.from_product([[1, 0], range(1, 10000)])

In [7]: mi.get_loc((1, 1))
Out[7]: 0

Problem description

The two engines should give the same result.

Expected Output


Output of pd.show_versions()


commit: f745e52
python-bits: 64
OS: Linux
OS-release: 4.9.0-3-amd64
machine: x86_64
byteorder: little
LC_ALL: None

pandas: 0.22.0.dev0+241.gf745e52e1.dirty
pytest: 3.2.3
pip: 9.0.1
setuptools: 36.7.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.0dev
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.0
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1


This comment has been minimized.


gfyoung commented Nov 27, 2017

Complex dtypes are probably not pandas cup of tea. That being said, we should still try to make our hashtables more robust. Have a look around and see what you can find.


This comment has been minimized.


jreback commented Nov 27, 2017

complex dtypes are pandas cup of tea, by-definition a tuple IS mixed dtype. so this is for sure in scope.


This comment has been minimized.


gfyoung commented Nov 27, 2017

complex dtypes are pandas cup of tea, by-definition a tuple IS mixed dtype

Okay, good to know. I was under a different impression as you can tell.


This comment has been minimized.


toobaz commented Nov 27, 2017

Well, I guess that in this specific case, the fact that the tuple is "complex" or not should be irrelevant, because it is just a container for values that refer to different levels - and the levels are homogeneous and nice, and the values are of the desired dtypes. So yes, this should be pretty standard.

(As opposed, for instance, to a tuple as a key for a flat index - which, ironically, works just fine)

@toobaz toobaz referenced this issue Jan 4, 2018


REF: codes-based MultiIndex engine #19074

3 of 3 tasks complete

@jreback jreback added this to the 0.23.0 milestone Jan 17, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment