Glacier archive fails with InvalidSignatureException on second attempt within AWSAuthConnection._mexe (multi-execute)

[phunter]

There seems to be an issue where the signature gets calculated incorrectly on the second attempt of a glacier archive POST request. This occurs in connection.py withiin AWSAuthConnection._mexe() in the rare event that the POST request fails on the first attempt.

I haven't dug in far enough to figure out how to fix this, but my hunch is that an extra item is getting stored in the connection header (possibly the 'User-Agent' key?) whcih then somehow gets passed into the canonical_request and string_to_sign. I don't have enough of an understanding of how the connection pooling and requst signing happen to localize this bug further at the moment, but I am willing to provide further details if someone call guide me in what to look for.

[jamesls]

I'm trying to reproduce this but I'm not able to. Do you know specifically how the POST is failing? Is the connection getting reset? Timeouts?

Is there a specific method you are calling that generates the InvalidSignatureException?

And also, if you have any debug logs you can share that should show us if there's any additional headers being added that shouldn't be.

[phunter]

The simplest way I've found to reproduce the error is to make _mexe think it has failed the first time. This is rather crude, but try changing line 852 of boto/connection.py from:

if response.status == 500 or response.status == 503:

to:

if i == 0 or response.status == 500 or response.status == 503:

Here is the code I'm using to do the archiving:

import os
import boto

AWS_ACCESS_KEY_ID = 'MY_AWS_ACCESS_KEY_ID'
AWS_SECRET_ACCESS_KEY = 'MY_AWS_SECRET_ACCESS_KEY'
GLACIER_VAULT = 'my_vault_name'

glacier = boto.connect_glacier(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
vault = glacier.get_vault(GLACIER_VAULT)

file_name = '/tmp/testfile.txt'
fp = open(file_name, 'w')
fp.write("hello glacier")
fp.close()

vault.upload_archive(file_name)
os.remove(file_name)

The error I get looks like:

boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 201, got (403, code=InvalidSignatureException, message=The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. ...)

Let me know what else I can do to help!

[phunter]

Ok, I think I might have tracked this far enough that I can identify the source of the problem. I don't think it has anything to do with headers as I originally thought. I was trying to get you some debug logs that isolated the issue and I noticed an unexplained difference between the Canonical Request fields. Notice that they look identical except for the last line.

First attempt:

DEBUG:boto:CanonicalRequest:
POST
/-/vaults/my_vault_name/archives

host:glacier.us-east-1.amazonaws.com
x-amz-content-sha256:fb847ff5db2a4fa46ba71380f37a8acd8320468f2aced210b3a586c8dc795e36
x-amz-date:20121217T233800Z
x-amz-glacier-version:2012-06-01
x-amz-sha256-tree-hash:fb847ff5db2a4fa46ba71380f37a8acd8320468f2aced210b3a586c8dc795e36

host;x-amz-content-sha256;x-amz-date;x-amz-glacier-version;x-amz-sha256-tree-hash
fb847ff5db2a4fa46ba71380f37a8acd8320468f2aced210b3a586c8dc795e36

Second attempt:

DEBUG:boto:CanonicalRequest:
POST
/-/vaults/my_vault_name/archives

host:glacier.us-east-1.amazonaws.com
x-amz-content-sha256:fb847ff5db2a4fa46ba71380f37a8acd8320468f2aced210b3a586c8dc795e36
x-amz-date:20121217T233800Z
x-amz-glacier-version:2012-06-01
x-amz-sha256-tree-hash:fb847ff5db2a4fa46ba71380f37a8acd8320468f2aced210b3a586c8dc795e36

host;x-amz-content-sha256;x-amz-date;x-amz-glacier-version;x-amz-sha256-tree-hash
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

That last line is actually the encrypted payload. In this case, the encryption takes place in utils in the compute_hash() method. It appears that since that function does an fp.read() of the file pointer passed in, subsequent times through the function, the file pointer needs to be reset to the beginning of the file. I added a fp.seek(0) right at the start of compute_hash() and my error went away.

Maybe you can tell me whether this solution is sufficient or whether there is something strange about the way the file is being initially opened that needs to be fixed.

[jamesls]

Wow, that's some very detailed debugging, much appreciated!

Looking at compute_hash, it appears to .tell() the current offset so that it can .seek() back to it after computing the sha256 hash. I don't think that anything in boto depends on that behavior, but given it's a public function in a utils module, there might be other code relying on this offset-preserving behavior.

This checksum is used for the sigv4 signing, so it might be better to put this in HmacAuthV4Handler.payload. I can put together a unittest/fix for this.

Thanks again.

[phunter]

Thanks for the quick response. Yes, I realized soon after I posted that there was a .tell() and .seek() in the function so my "fix" was in fact rather sloppy. Glad you have a better solution in mind!

[jamesls]

After trying to implement this, I'm not sure my previous suggestion of putting it in the auth handler was good. I think the two best options would be:

• Add a reset() method to the HTTPRequest object that _mexe calls when it needs to retry a request.
• Add a custom sender callable to make_request that resets the fileobj before it's called (this is what the s3 module does to account for this).

The second option would be nice because this could fix #971 as well. It also isolates this logic to the glacier module, which is one of the only modules using fileobjs for the body param.

[jamesls]

I ended up implementing the second option, given that it's a more isolated change. Unfortunately, I couldn't come up with a good way to also fix #971 so I'll have to figure something else out for that (I'll add a note in #971).

Thanks again for the report.