AWS error response when request date does not match signing key date

[sam-washington]

Description

Reported by @ipartola in pull request #3.

When the date of a request, as given in either the Date or the X-Amz-Date header, is different from the date in the AWS4Auth object's signing key scope, then an invalid signature is generated and AWS sends an authentication error in response to the request.

This issue affects request-aws4auth versions 0.7 and earlier.

As descibed by @ipartola:

"The bug is that the timestamp for the signing key is generated when the object is instantiated. Once we cross a date boundary it seems AWS requires that the signing key's scope date match the x-amz-date header. The solution I propose is to continually set the current date for the signing key to match the x-amz-date header."

Discussion

The AWS documentation for date handling in signing key scopes is here: http://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

It could be argued that this isn't a bug in requests-aws4auth, since it should be up to the calling code to ensure that the scope of the signing key is kept up to date by generating a new AWS4Auth instance. But it's obvious that this is a common situation and that it would be a very useful feature if requests-aws4auth could handle this automatically.

Allowing the key to automatically regenerate raises problems for sharing an AWS4Auth instance between multiple threads, since if the key can be regenerated automatically at any time then it could be modified by another thread while it is being used to sign a request.

It also means that the AWS secret key will need to be stored in the signing key, so that the key can be regenerated without having to escape back out to the code generating the request.

Proposed Solution

As suggested by @ipartola, the key needs to be regenerated with a date that matches the date in the request.

As this is likely to be a common situation, and a bit of a WTF moment when new users hit it, it is proposed that automatic key regeneration is made the default behaviour for new instances of AWS4Auth. However, the behaviour needs to be made configurable, for situations where people want to share instances between threads, or don't want the key stored in the instance, or don't want automatic regeneration for other reasons. Retaining a way of duplicating the current behaviour (i.e. signing the request with a bad signature) is also desirable for backward compatibility if needed.

The proposal is to modify the AWS4Auth class to check a request's X-Amz-Date or Date headers against the scope date, and if it is different then automatically regenerate the signing key with a scope that uses the new date. This means when new instances of AWS4Auth are created, the AWS secret key will need to be supplied.

A new method will be added to AWS4Auth: `regenerate_signing_key()``. Calling this will trigger an immediate regeneration of the instance's signing key. Parameters can be supplied to this method to set new values for the key scope parameters: region, service and date. This method will be used when a automatic key regeneration is needed, but can also be called by other code if a manual regeneration is needed.

Two new subclasses of AWS4Auth will be added: StrictAWS4Auth and PassiveAWS4Auth. StrictAWS4Auth will not regenerate the signing key when a date mismatch is encountered, it will instead raise a new exception, DateMismatchError. This will allow the calling code to handle the situation itself if desired.

PassiveAWS4Auth will do no date checking, and will sign the request and send it even if the request date does not match the scope date. This mimics the current behaviour of v0.7 AWS4Auth.

Further, a new parameter will be added to AWS4SigningKey: store_secret_key. If this is set to True, which will be the default, the secret key is stored in the instance, not thrown away as it is in v0.7. By creating instances of AWS4Auth or its subclasses using AWS4SigningKey instances created with store_secret_key set to False, you can control whether the secret key is stored in an AWS4Auth instance or not.

Examples

Regular usage, usual case with no date change, X-Amz-Date header is automatically added to the request by the AWS4Auth signing process, using today's date, since Requests doesn't automatically add a date header. This is the same usage and behaviour as current 0.7 version when not crossing a date boundary:

>>> auth = AWS4Auth('access_id', 'secret_key', 'us-east-1', 's3')
>>> auth.date
'20151222'
>>> response = requests.get('http://s3.amazonaws.com', auth=auth)
>>> response.text
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01">
  <Owner>
    <ID>bcaf1ffd86f461ca5fb16fd081034f</ID>
    <DisplayName>webfile</DisplayName>
>>> response.requests.headers['X-Amz-Date']
'20151222T212208Z'

Next this shows regular usage, but crossing a date boundary and AWS4Auth automatically regenerating the signing key. This is the main use case for this fix. (In this case the X-Amz-Date header is again added automatically by the AWS4Auth authentication using the current date. It is added before the scope/request date check is made, and can be different to the scope date):

>>> auth = AWS4Auth('access_id', 'secret_key', 'us-east-1', 's3')
>>> auth.date
'20151222'
>>> id(auth.signing_key)
11162416
...wait until tomorrow...
>>> datetime.utcnow().strftime('%Y-%m-%d')
'2015-12-23'
>>> response = requests.get('http://s3.amazonaws.com', auth=auth)
>>> response.text
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01">
  <Owner>
    <ID>bcaf1ffd86f461ca5fb16fd081034f</ID>
    <DisplayName>webfile</DisplayName>
>>> response.request.headers['X-Amz-Date']
'20151232T000934Z'
>>> auth.date
'20151223'
>>> id(auth.signing_key)
10947184

Next is an example with StrictAWS4Auth. With this subclass the key is not automatically regenerated, instead a DateMismatchError is raised when the request date does not match the signing key scope date:

>>> auth = StrictAWS4Auth('access_id', 'secret_key', 'us-east-1', 's3')
>>> auth.date
'20151222'
>>> id(auth.signing_key)
11162416
...wait until tomorrow...
>>> datetime.utcnow().strftime('%Y-%m-%d')
'2015-12-23'
>>> response = requests.get('http://s3.amazonaws.com', auth=auth)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  ...
requests_aws4auth.exceptions.DateMismatchError

The key can now be manually regenerated to match the new day's date:

>>> today = datetime.utcnow().strftime('%y%m%d')
>>> today
'20151223'
>>> auth.regenerate_signing_key(date=today)
>>> auth.date
'20151223'
>>> id(auth.signing_key)
10947184

Example again using StrictAWS4Auth, but also where the secret key is not stored in the auth instance:

>>> sig_key = AWS4SigningKey('secret_key', 'us-east-1', 's3', store_secret_key=False)
>>> auth = AWS4Auth('access_id', sig_key)
>>> auth.date
'20151222'
>>> id(auth.signing_key)
11162416
...wait until tomorrow...
>>> datetime.utcnow().strftime('%Y-%m-%d')
'2015-12-23'
>>> response = requests.get('http://s3.amazonaws.com', auth=auth)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  ...
requests_awsauth.exceptions.DateMismatchError
>>> today = datetime.utcnow().strftime('%y%m%d')
>>> auth.regenerate_signing_key(date=today)
requests_aws4auth.exceptions.NoSecretKeyError

Here the manual regeneration fails because there is no stored secret key. To regenerate, the secret key must be passed to regenerate_signing_key():

>>> auth.regenerate_signing_key(secret_key='secret_key', date=today)
>>> auth.date
'20151223'
>>> id(auth.signing_key)
10947184

The key is not stored in the new key either, because the new key uses the value of store_secret_key in the old key, which isFalse in this case.

Any feedback on these proposed changes gratefully received.

[sam-washington]

Code changes to implement this commited to the 0.8_dev branch. Take a look and let me know if you have any problems or thoughts. There are more details on the implementation in the updated module docstrings.

I'll leave this out there for testing and comment until 30th Dec, then I'll merge to master and release on PyPi.

[ipartola]

Just gave this a try and it seems to be working. Thank you for all the work.

[sam-washington]

Version 0.8 has been published to PyPi, the 0.8_dev branch has been merged to master and tagged 0.8.

[rahulmuraliqllabscom]

Can any one tell me how to add data in the request. I am getting signature mismatch while adding data

[sam-washington]

Hi @rahulmuraliqllabscom, are you still having trouble with this? Can you post some sample code?

[rahulmuraliqllabscom]

Hi Sam I just went through the aws signing tutorial and wrote a code snippet myself. Thanks for showing concern.