UnicodeDecodeError: 'ascii' codec can't decode byte

[ShiftyMcCool]

Hello,

I've searched for this error and have found several closed issues for this error, but I am getting the following error:

Traceback (most recent call last):
  File "/webroot/common-content/new_lambda_function.py", line 52, in lambda_handler
    index_response = es.index(index="resources", doc_type="resource", id=resource['content']['id'], body=resource['content'])
  File "/webroot/common-content/venv/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/webroot/common-content/venv/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 279, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/webroot/common-content/venv/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/webroot/common-content/venv/local/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 68, in perform_request
    response = self.session.request(method, url, data=body, timeout=timeout or self.timeout)
  File "/webroot/common-content/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/webroot/common-content/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/webroot/common-content/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/webroot/common-content/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/webroot/common-content/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 979, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 833, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 568: ordinal not in range(128)

It seems that a previously posted workaround has been implemented (#374). This issue report is my problem exactly, is there any solution to this with Python 2.7? There is no chance of converting to Python 3 since AWS Lambda does not support it.

Thanks!

[honzakral]

The issue is that you are using non-ascii bytestrings in python 2 while having your default encoding set to ascii. When then python tries to serialize that data, it fails because the configuration doesn't match the data.

The proper fix is to use unicode type instead of str or set the default encoding properly to (I assume) utf-8.

Hope this helps

[cristobalcl]

I fix this isuue with this hack: http://stackoverflow.com/a/17628350/482177

Any better way to fix it?

[honzakral]

That is not a hack, if you are using utf-8 you should have your encoding set to utf-8. Better way to fix it would be, however, to not use str in python 2 for representing non-ascii data and use unicode instead.

[sddj]

I know this is old and dead, but I ran into this issue when using requests_aws4auth==0.9, and I thought it might be helpful to share what I found. The aws4auth.py file includes 'from future import unicode_literals', which makes all of the strings in the file unicode instances, including the strings used for HTTP header keys. Python's httplib puts all the headers together into the msg variable above, and since merging unicode instances with str instances results in a unicode instance, msg is a unicode. The httplib code is assuming that msg is a str instance (since msg at this point is just the HTTP request header and should be ASCII). It then tries to append the request body (which is a str that has already been encoded into the charset specified in the Content-Type header) to the request header (which should be a str but is a unicode because of aws4auth.py), and you get an error. I'm not sure really who is at fault here, but I think the aws4auth.py should not be setting unicode HTTP headers.

[beliefless-dogmasless]

Recompile python27, with --enable-unicode

[kapilt]

i ran into this with the bulk indexer helpers, for python2.7 the elasticsearch library is explicitly joining non unicode strings causing the implicit conversion and reliance on default system encoding.

ie. from elasticsearch/helpers/init.py

        resp = client.bulk('\n'.join(bulk_actions) + '\n', *args, **kwargs)  

the actual serialization of bulk actions is performed by the transport serializer, using the RequestsHttpConnection as transport the json serialization returns a unicode object, which combined with this byte string join is the trigger on the unicode conversion and decode errors.

[honzakral]

@kapilt those are actually unicode objects since the file is using from __future__ import unicode_literals

The tl;dr of this issue is that if you are using non-ascii data you must use unicode or delare default encoding properly, there is just no way around that.

[paperlessmichael]

For anyone stumbling upon this, see #374 for a solution, as mentioned above.