Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add encode to support non-English text data #75

Merged
merged 4 commits into from
Nov 20, 2019
Merged

Add encode to support non-English text data #75

merged 4 commits into from
Nov 20, 2019

Conversation

muziyoshiz
Copy link
Contributor

When I call awscurl 0.17 with non-English text data, it fails with the following UnicodeEncodeError.

$ awscurl -XGET --service es "${ENDPOINT}/${INDEX}/_search" -d '日本語'
Traceback (most recent call last):
  File "/usr/bin/awscurl", line 11, in <module>
    load_entry_point('awscurl==0.17', 'console_scripts', 'awscurl')()
  File "/usr/lib/python3.6/dist-packages/awscurl/awscurl.py", line 395, in main
    args.insecure
  File "/usr/lib/python3.6/dist-packages/awscurl/awscurl.py", line 220, in make_request
    return __send_request(uri, data, headers, method, verify)
  File "/usr/lib/python3.6/dist-packages/awscurl/awscurl.py", line 244, in __send_request
    r = requests.request(method, uri, headers=headers, data=data, verify=verify)
  File "/usr/lib/python3.6/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1299, in _send_request
    body = _encode(body, 'body')
  File "/usr/lib64/python3.6/http/client.py", line 171, in _encode
    (name.title(), data[err.start:err.end], name)) from None
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 126-127: Body ('日本語') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

It is because the data passed to requests.request method is not encoded.
Please check this pull request. Thanks.

@okigan
Copy link
Owner

okigan commented Nov 19, 2019

Thanks -- Please add a unit test

@muziyoshiz
Copy link
Contributor Author

@okigan
Thanks for your review!
I added the integration tests both for English data and non-English data. Please check.

Copy link
Owner

@okigan okigan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@okigan okigan merged commit 65a3c29 into okigan:master Nov 20, 2019
@muziyoshiz
Copy link
Contributor Author

Thank you!

@muziyoshiz
Copy link
Contributor Author

@okigan
Would you please let me know when the next release will be?
Now I am using awscurl v0.17 with my patch. It is good to install the latest master version with pip for non-English users like me. Thanks.

@okigan
Copy link
Owner

okigan commented Nov 22, 2019

next day or two, thanks!

@okigan
Copy link
Owner

okigan commented Dec 3, 2019

"next day or two" did not account for the thanksgiving week -- hope everyone had a good break as v0.18 is now released and up on pypi, happy awscurling!

@muziyoshiz
Copy link
Contributor Author

I see 😅 No problem. Thank you very much!

okigan added a commit that referenced this pull request Dec 21, 2019
Add encode to support non-English text data
okigan added a commit that referenced this pull request Dec 21, 2019
Add encode to support non-English text data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants