Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect sigv4 signature when non-ascii present in query string in Python 2 #2846

Closed
pslawski opened this issue Dec 30, 2014 · 2 comments
Closed

Comments

@pslawski
Copy link
Contributor

When using sigv4, HTTP requests to S3 are incorrectly signed when non-ascii characters are present in the query string in Python 2. When sigv4 auth is added to a HTTP request for S3, the auth path and parameters are mangled. The auth path's query string is removed, parsed, and placed into the request parameters. The updated requests parameters are then used for part of the signature calculations. Parsing of the query string is done using urllib.parse.parse_qs. In Python 2, parse_qs does not decode URL-encoded %HH escapes. Thus, it outputs wrongly if given a Unicode query string with URL-encoded non-ascii characters.

>>> urllib.parse.parse_qs(u'prefix=El%20Ni%C3%B1o') 
{u'prefix': [u'El Ni\xc3\xb1o']} 

>>> u'El Ni\xc3\xb1o' == 'El Ni\xc3\xb1o'.decode('utf-8')
False  # parse_qs did not decode to utf-8 in Python 2

>>> 'El Ni\xc3\xb1o'.decode('utf-8') == u'El Ni\xf1o'
True

In the below log snippet, you can see this in action when the request path does not agree with the CanonicalRequest parameters:

HTTP request query string: max-keys=1&prefix=El%20Ni%C3%B1o/
CanonicalRequest Params: max-keys=1&prefix=El%20Ni%C3%83%C2%B1o%2F

  File "/usr/lib/python2.6/site-packages/boto-2.34.0-py2.6.egg/boto/s3/bucket.py", line 475, in get_all_keys
    '', headers, **params)
  File "/usr/lib/python2.6/site-packages/boto-2.34.0-py2.6.egg/boto/s3/bucket.py", line 413, in _get_all
    response.status, response.reason, body)
S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
...
boto: DEBUG: Method: GET
boto: DEBUG: Path: /?max-keys=1&prefix=El%20Ni%C3%B1o/
boto: DEBUG: Data: 
boto: DEBUG: Headers: {}
...
boto: DEBUG: CanonicalRequest:
GET
/
max-keys=1&prefix=El%20Ni%C3%83%C2%B1o%2F
...
@pslawski
Copy link
Contributor Author

pslawski commented Feb 6, 2015

Opened a pull request to resolve this, #2844

@jamesls
Copy link
Member

jamesls commented Apr 4, 2016

Closing, #2844 has been merged.

@jamesls jamesls closed this as completed Apr 4, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants