RFC for AWS SignatureVersion 4 for EC2

[gertjanol]

As shortly discussed with @Kami in #407, this patch adds support for Signature Version 4 to the SignedAWSConnection class. v4 will be used when a Driver-class selects EC2V4Connection as its connectionCls. In this patch this is only enabled for the the new Frankfurt region, represented by a new EC2EUCentralNodeDriver class.
Haven't created any tests yet, since this is just the PoC. If this has any chance of getting merged in, I'll do the tests.

I would like some feedback and thoughts on this work.
Some questions:

• Is this the way to go, with the Connection-classes specifying a version, and the Driver classes specifying the Connection class to use?
• I've chosen the pre_connect_hook to do the work, because we need both the headers and the parameters to calculate the signature. Hope that's ok?
• I’ve tested all regions for EC2 and all seem to work fine with the v4 authentication. I haven’t tested the only other part that uses the SignedAWSConnection, the ELBConnection, but I’m guessing that that also supports V4. It would reduce complexity if we could drop support for V2 in the SignedAWSConnection altogether.
• The credential scope, as explained here needs the service we are requesting. This should be just ‘ec2’ for compute service. I found no existing way to retrieve this, so I introduced a new class variable in the EC2Connection. I don’t like this, so maybe there is a better way?
• This patch now only works for GET-requests, but the EC2-driver doesn't use anything else. I think S3 uses other methods, but seems to have its own authentication routines. Why is that? Should I take other methods into consideration here?
• Performance of calculating this signature sucks :(. Haven't done any measuring, but v2 seems way faster.

[Kami]

Great, thanks - I will look asap.

[Kami]

Hm, is this correct? It looks like it's just calculating SHA256 hash of an empty string.

[gertjanol]

Yeah, one of the reasons this patch only works for GET-requests. This should be a hash of the payload of the request, which for GET requests is empty. See http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html, step 6.

[gertjanol]

@Kami what do you think of this? I'm happy to improve this work, but would like some feedback first.

[Kami]

@gertjanol So far it looks fine, but I didn't have time to test it end to end yet.

[allardhoeve]

datetime.utcnow returns a naive datetime. You should use datetime.now(tzinfo=libcloud.utils.uso8601.UTC).

https://github.com/apache/libcloud/blob/trunk/libcloud/utils/iso8601.py#L65

[allardhoeve]

But as this currently works, I deduce that AWS is not that strict about this header 😉

[allardhoeve]

I'm loving it, even though we're only field-tested for EC2. But, that's exactly why there is only the EC2Connection class and no others. 👍

@Kami?

[Kami]

@allardhoeve You tested it with the new Frankfurt region?

[gertjanol]

@Kami we're using this right now for both Frankfurt and Ireland region (@allardhoeve and me == co-workers :))

[Kami]

Cool - I will also look into it (and test it) myself and try to get it merged ASAP.

[Kami]

I just noticed an issue while testing this change - this code will only use signature v4 if you are using old and deprecated class based approach. It won't use it if you use the region argument.

EC2 = get_driver(Provider.EC2)
driver = EC2('...', '...', region='eu-central-1')

 -------- begin 140518062231056 request ----------
curl -i -X GET -H 'Host: ec2.eu-central-1.amazonaws.com' -H 'X-LC-Request-ID: 140518062231056' -H 'Accept-Encoding: gzip,deflate' -H 'User-Agent: libcloud/0.17.1-dev (Amazon EC2) ' --compress 'https://ec2.eu-central-1.amazonaws.com:443/?SignatureVersion=2&AWS...'

I will work on a fix.

[Kami]

I will probably do something along those lines:

1. Modify SignedAWSConnection class to take signature_version argument in the constructor and default it to 2.
2. Modify _ex_connection_class_kwargs method to pass the correct version to the connection class constructor in case the region is eu-central-1
3. Remove the new `EC2EUCentralNodeDriver``class since the class based approach has been deprecated quite a while ago now (see https://libcloud.readthedocs.org/en/latest/upgrade_notes.html#amazon-ec2-compute-driver-changes).

@gertjanol Will change nr. 3 affect you? Do you use a class per region approach?