Support S3 authentication via IAM Role credentials

[cyberduck]

6c94084 created the issue

Would it be possible to add support for authentication to AWS S3 using Access Key credentials that are derived from an IAM Role on an EC2 instance?

IAM Roles allow you to assign a set of permissions to a resource that is actually deployed in the AWS environment. The way this is implemented is that a set of credentials (Access Key + Secret Key + Session Token) are dynamically assigned and rotated to a particular AWS resources, such as an EC2 Instance.

The Access Key credentials can be retrieved on an EC2 instance by accessing its own instance meta data via the URL:

http://169.254.169.254/latest/meta-data/iam/security-credentials/THIS_IS_THE_ROLE_NAME

The last component of that URL path (THIS_IS_THE_ROLE_NAME) is actually the name of the role assigned, but it is the only entry in that path, so to get to it programmatically, you would need to call its parent URL and find the only returned value in the response, and then retrieve the target URL which contains the actual credentials.

The actual credentials returned look like the following JSON fragment:

{
  "Code" : "Success",
  "LastUpdated" : "2015-02-24T21:10:36Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "EQWFTCASIAJ3KZUVUTYA",
  "SecretAccessKey" : "wli/Bu889nQjdRxpgF6QR3Hoqjz8Lou7pnoxBU/r",
  "Token" : "AQoDYXdzEN_//_////wEa0AMBFkIFLfF7oMkGW9MnnPwNuRNikTvwPiHCk+DOrQgIZ9/vas0tUoe35TPBbnMB6keqO0IZaFvwICaA4k83X+clJ3aRC+E26K6oC8H3LDTfEFWofnoxuPGCq8MKPdMJOz2URnwn0Mv5p4ecuxXmNJqD2pdh65xH7jtA4slK3ZyD6ggXvSSMzlq9VeVSTz39V57FZNRQ6u4JcjWv3gBqfrJyTY10uLP8N4xMO0M7uEU9hnXlwxbMKkm3arjjl81IVYTh4OIaju3NcCsBMnWqxetsZtCWG4+SZbT6RWKOZMD7r8joIsjomRIkuJDjSpVL/f3Xa1iphF+qOFVsYUav2XNQukvHcborOWy2CIDBnOsMrA67z8BVByZG6qLQBywsxTzYv/w+nEWkY1avVWAhcWeMxDrx3UYVtHLnObZUdhnEbeXrTgwbjXqYRFi4Pe8cnMNRQvcmh08MPGEiXLtqc7G0zIr8yWdl0Im5CK4OBSkPAzujkcnu6hjMAaVkuXu+OPU9Q/qV9mLx6C8+VhZ/udPY2537qCGL3H9/2LWaLO9uCrRYpx+f1es+idg12P+bO68aN0G6mzKDbGZsDjJqONt5622CYDZBWJumAlFQYmTcmSCX0bOnBQ==",
  "Expiration" : "2015-02-25T03:16:33Z"
}

From this response you can get the following 3 components which are needed to authenticate requests to the AWS S3 API:

• AccessKeyId - the Access Key
• SecretAccessKey - the Secret Key
• Token - the Session Token

You can also obtain the LastUpdated and Expiration components which indicate when the credentials were last generated, as well as when they will expire and be rotated once again.

For CyberDuck, the request I'm making is to add an option flag to the S3 connection settings to indicate the use of IAM Role credentials (similar to the existing "Anonymous" flag). When set, CyberDuck would obtain the current Role credentials as described above, and store them in the current user session, and with every API call to S3, verify the creds are still valid (by comparing current time against the expected Expiration time), and use those creds to authenticate/authorize each API request.

This will allow CyberDuck to be used on an EC2 instance within AWS assuming the IAM Role access policies.

[cyberduck]

6c94084 commented

After taking another look at the "Open Connection" and "Add Bookmark" dialog boxes, I see that the actual structure of these dialogs is pretty much fixed across all the different providers, and the only thing that varies, is the labeling of a few fields (cred-related) and also the options that are available in the "Connect Mode" drop-down.

To keep consistent with this approach, this feature could be implemented either (1) as a completely new provider type (S3 IAM Role) or (2) as a drop-down selection for the "Connect Mode" option under the S3 provider.

[cyberduck]

@dkocher commented

Refer to IAM Roles for Amazon EC2.

[cyberduck]

@dkocher commented

Do you run Cyberduck on a Windows EC2 instance?

[cyberduck]

@dkocher commented

This feature would also be very suitable when running Cyberduck CLI on a Linux instance.

[cyberduck]

@dkocher commented

We would set this up with a Profile required to be installed.

[cyberduck]

6c94084 commented

Replying to [comment:3 dkocher]:

Do you run Cyberduck on a Windows EC2 instance?

Yes, we're rolling out a few intances in AWS right now and each one is assigned an IAM Role specific to their function or service class. Those roles determine permissions the instances inherit to support their functions wrt/ calls and access to different AWS services and resources, including S3 where the different instances have access to different buckets and actions.

[cyberduck]

6c94084 commented

I was reviewing some of the relevant code to implement this request, and I thought of another way to add the functionality with minimal disruption to the existing UI or codebase as starting point.

Instead of touching any of the UI elements at all, a special sentinel value, such as %IAM_ROLE%, that when provided for the "Access Token" of an S3 connection (i.e. the Username), triggers slightly different behavior in the S3 connection building.

Looking at the latest version of the code base in trunk for S3Session.java (source/ch/cyberduck/core/s3/S3Session.java @ 64deab7), at line 114 where you handle switching between an anonymous connection (null AWSCredentials) or constucting a set of credentials based on user-provided Access Key + Secret Key, you could add a third option when IAM_ROLE_SENTINEL.equals(host.GetCredentials().GetUsername()) to construct an AWSSessionCredentials instance.

The Access Key, Secret Key and Session Token would all be derived from the running context as described above.

[cyberduck]

@dkocher commented

In a7868f8. Untested besides unit tests. Please install the S3 (Temporary Credentials).cyberduckprofile profile and give it a try with the latest snapshot build.

[cyberduck]

@dkocher commented

Documentation in Connecting with temporary access credentials from EC2.

[cyberduck]

@dkocher commented

Add option to override username and password requirements in profile to make it skip the login prompt in 49f5f32.

[cyberduck]

@dkocher commented

We have verified this works with Cyberduck CLI on EC2. Change the default role name s3access in the Context URI in the profile accordingly. Defaults to http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access.

[ec2-user@ip-172-30-0-79 ~]$ duck --list s3-role://cyberduck-sandbox/ -v
Authenticating as anonymous…
> GET /latest/meta-data/iam/security-credentials/aws-ec2-role HTTP/1.1
> Host: 169.254.169.254
> Connection: Keep-Alive
> User-Agent: Cyberduck/4.7 (Linux/3.14.27-25.47.amzn1.x86_64) (amd64)
> Accept-Encoding: gzip,deflate
< HTTP/1.0 200 OK
< Content-Type: text/plain
< Accept-Ranges: bytes
< ETag: "1762412530"
< Last-Modified: Fri, 27 Feb 2015 22:01:56 GMT
< Content-Length: 898
< Connection: keep-alive
< Date: Fri, 27 Feb 2015 22:09:28 GMT
Listing directory cyberduck-sandbox…

[cyberduck]

6c94084 commented

We verified this works as designed, thank you!

Sorry for the delay, we did not expect you would have addressed this ticket so fast and made it available in a short-term release, so it took us a bit to be able to test it.