Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Druid + Hive query "22" error #7520

Open
tvorogme opened this issue May 15, 2019 · 1 comment

Comments

Projects
None yet
1 participant
@tvorogme
Copy link

commented May 15, 2019

To reproduce this issue you will need to create external table stored by druid in hive and try to query it from superset.

In sql lab you will see smth like this:

latest-screenshot

The real error is here:

Traceback (most recent call last):
  File "/home/tvorogme/projects/incubator-superset/superset/sql_lab.py", line 196, in execute_sql_statement
    data = db_engine_spec.fetch_data(cursor, query.limit)
  File "/home/tvorogme/projects/incubator-superset/superset/db_engine_specs.py", line 1400, in fetch_data
    return super(HiveEngineSpec, cls).fetch_data(cursor, limit)
  File "/home/tvorogme/projects/incubator-superset/superset/db_engine_specs.py", line 160, in fetch_data
    return cursor.fetchall()
  File "/usr/lib/python3.7/site-packages/pyhive/common.py", line 136, in fetchall
    return list(iter(self.fetchone, None))
  File "/usr/lib/python3.7/site-packages/pyhive/common.py", line 105, in fetchone
    self._fetch_while(lambda: not self._data and self._state != self._STATE_FINISHED)
  File "/usr/lib/python3.7/site-packages/pyhive/common.py", line 45, in _fetch_while
    self._fetch_more()
  File "/usr/lib/python3.7/site-packages/pyhive/hive.py", line 389, in _fetch_more
    schema = self.description
  File "/usr/lib/python3.7/site-packages/pyhive/hive.py", line 320, in description
    type_code = ttypes.TTypeId._VALUES_TO_NAMES[type_id]
KeyError: 22

The problem is in connector.

Lets see what is ttypes.TTypeId._VALUES_TO_NAMES:

{0: 'BOOLEAN_TYPE', 1: 'TINYINT_TYPE', 2: 'SMALLINT_TYPE', 3: 'INT_TYPE', 4: 'BIGINT_TYPE', 5: 'FLOAT_TYPE', 6: 'DOUBLE_TYPE', 7: 'STRING_TYPE', 8: 'TIMESTAMP_TYPE', 9: 'BINARY_TYPE', 10: 'ARRAY_TYPE', 11: 'MAP_TYPE', 12: 'STRUCT_TYPE', 13: 'UNION_TYPE', 14: 'USER_DEFINED_TYPE', 15: 'DECIMAL_TYPE', 16: 'NULL_TYPE', 17: 'DATE_TYPE', 18: 'VARCHAR_TYPE', 19: 'CHAR_TYPE', 20: 'INTERVAL_YEAR_MONTH_TYPE', 21: 'INTERVAL_DAY_TIME_TYPE'}

Ok. Now we need to know where type_id coming from:

primary_type_entry = col.typeDesc.types[0]
type_id = primary_type_entry.primitiveEntry.type

And now if you want to see column - it will be __time

So i have created issue in PyHive.

The easy solution for now replace 313 line in pyhive/hive.py

from

type_code = ttypes.TTypeId._VALUES_TO_NAMES[type_id]
to
type_code = ttypes.TTypeId._VALUES_TO_NAMES[type_id if type_id != 22 else 18]

18 is datetime so all will be cool:

latest-screenshot

P.S. I know that is not superset problem, but i guess many superset users use pyhive, so I just wanted to share my solution.

@issue-label-bot issue-label-bot bot added the #bug label May 15, 2019

@issue-label-bot

This comment has been minimized.

Copy link

commented May 15, 2019

Issue-Label Bot is automatically applying the label #bug to this issue, with a confidence of 0.97. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.