How to send a unicode query?

[arikfr]

I'm using a code similar to the following:

query = u"SELECT 'こんにちは'"

connection = tdclient.connect(apikey="...", type="presto", db="...")
cursor = connection.cursor()
cursor.execute(query.encode('utf8'))

And getting the following exception:

'ascii' codec can't decode byte 0xe3 in position 92: ordinal not in range(128)

This is the stack trace:

Traceback (most recent call last):
  File "query.py", line 107, in run_query
    cursor.execute(query.encode('utf8'))
  File "/Users/arik/.virtualenvs/rd/lib/python2.7/site-packages/tdclient/cursor.py", line 45, in execute
    self._executed = self._api.query(query, **self._query_kwargs)
  File "/Users/arik/.virtualenvs/rd/lib/python2.7/site-packages/tdclient/job_api.py", line 229, in query
    with self.post("/v3/job/issue/%s/%s" % (urlquote(str(type)), urlquote(str(db))), params) as res:
  File "/Users/arik/.virtualenvs/rd/lib/python2.7/site-packages/tdclient/api.py", line 220, in post
    response = self.send_request("POST", url, fields=fields, body=body, headers=headers, decode_content=True, preload_content=False)
  File "/Users/arik/.virtualenvs/rd/lib/python2.7/site-packages/tdclient/api.py", line 345, in send_request
    return self.http.request(method, url, fields=fields, **kwargs)
  File "/Users/arik/.virtualenvs/rd/lib/python2.7/site-packages/urllib3/request.py", line 70, in request
    **urlopen_kw)
  File "/Users/arik/.virtualenvs/rd/lib/python2.7/site-packages/urllib3/request.py", line 148, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/Users/arik/.virtualenvs/rd/lib/python2.7/site-packages/urllib3/poolmanager.py", line 244, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/arik/.virtualenvs/rd/lib/python2.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/Users/arik/.virtualenvs/rd/lib/python2.7/site-packages/urllib3/connectionpool.py", line 361, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1053, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1093, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 891, in _send_output
    msg += message_body

I've tried encoding it first (query.encode('utf-8')), but got the same error.

Am I doing something wrong?

[yyuu]

I confirmed that it's working if I specified query as a string instead of bytes. Because td-client-python can manage unicode string, you must not need to call encode on string in arguments.

% ipython
impPython 3.5.2 (default, Sep 23 2016, 07:55:24)
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
ort t
In [1]: import tdclient

In [2]: query = u"SELECT 'こんにちは'"

In [3]: connection = tdclient.connect(apikey="...", type="presto", db="sample_datasets")

In [4]: cursor = connection.cursor()

In [5]: cursor.execute(query)
Out[5]: '109757645'

In [6]: cursor.fetchmany()
Out[6]: [['こんにちは']]

[yyuu]

I confirmed the reproduction of UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 92: ordinal not in range(128) with 2.7.13....

[yyuu]

I opened #28 as a workaround for unicode encoding issue on Python 2.x. I hope it helps.

[yyuu]

@arikfr I've released 0.8.0.dev0 to PyPI including fixes in #28 and #29. I confirmed that it's running for my test queries including unicode characters. Please give it a try when you get a chance.

https://pypi.python.org/pypi/td-client/0.8.0.dev0

[yyuu]

I've released 0.8.0 including the fixes. Just upgrading to the latest release should solve the issue. Feel free to reopen is you still have some issues.

[arikfr]

Sorry for the late follow up - I can confirm that v0.8.0 fixed this. Thanks!