Skip to content

Latest commit

 

History

History
296 lines (227 loc) · 15.2 KB

NOTES.aws-auth.md

File metadata and controls

296 lines (227 loc) · 15.2 KB

AWS Authentication Implementation Notes

AWS Account

Per [its documentation](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html, the GetCallerIdentity API call that the server makes to STS to authenticate the user using MONGODB-AWS auth mechanism requires no privileges. This means in order to test authentication using non-temporary credentials (i.e., AWS access key id and secret access key only) it is sufficient to create an IAM user that has no permissions but does have programmatic access enabled (i.e. has an access key id and secret access key).

AWS Signature V4

The driver implements the AWS signature v4 internally rather than relying on a third-party library (such as the AWS SDK for Ruby) to provide the signature implementation. The implementation is quite compact but getting it working took some effort due to:

  1. The server not logging AWS responses when authentication fails
  2. Some of the messages from STS being quite cryptic (I could not figure out what the problem was for either "Request is missing Authentication Token" or "Request must contain a signature that conforms to AWS standards", and ultimately resolved these problems by comparing my requests to those produced by the AWS SDK).
  3. Amazon's own documentation not providing an example signature calculation that could be followed to verify correctness, especially since this is a multi-step process and all kinds of subtle errors are possible in many of the steps like using a date instead of a time, hex-encoding a MAC in an intermediate step or not separating header values from the list of signed headers by two newlines.

Reference Implementation - AWS SDK

To see actual working STS requests I used Amazon's AWS SDK for Ruby (API docs for STS client, configuration documentation) as follows:

  1. Set the credentials in the environment (note that the region must be explicitly provided):

    export AWS_ACCESS_KEY_ID=AKIAREALKEY export AWS_SECRET_ACCESS_KEY=Sweee/realsecret export AWS_REGION=us-east-1

  2. Install the correct gem and launch IRb:

    gem install aws-sdk-core irb -raws-sdk-core -Iaws/sts

  3. Send a GetCallerIdentity request, as used by MongoDB server:

    Aws::STS::Client.new( logger: Logger.new(STDERR, level: :debug), http_wire_trace: true, ).get_caller_identity

This call enables HTTP request and response logging and produces output similar to the following:

opening connection to sts.amazonaws.com:443...
opened
starting SSL for sts.amazonaws.com:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-SHA
<- "POST / HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded; charset=utf-8\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby3/3.91.1 ruby/2.7.0 x86_64-linux aws-sdk-core/3.91.1\r\nHost: sts.amazonaws.com\r\nX-Amz-Date: 20200317T194745Z\r\nX-Amz-Content-Sha256: ab821ae955788b0e33ebd34c208442ccfc2d406e2edc5e7a39bd6458fbb4f843\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAREALKEY/20200317/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=6cd3a60a2d7dfba0dcd17f9c4c42d0186de5830cf99545332253a327bba14131\r\nContent-Length: 43\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "x-amzn-RequestId: c56f5d68-8763-4032-a835-fd95efd83fa6\r\n"
-> "Content-Type: text/xml\r\n"
-> "Content-Length: 401\r\n"
-> "Date: Tue, 17 Mar 2020 19:47:44 GMT\r\n"
-> "\r\n"
reading 401 bytes...
-> ""
-> "<GetCallerIdentityResponse xmlns=\"https://sts.amazonaws.com/doc/2011-06-15/\">\n  <GetCallerIdentityResult>\n    <Arn>arn:aws:iam::5851234356:user/test</Arn>\n    <UserId>AIDAREALUSERID</UserId>\n    <Account>5851234356</Account>\n  </GetCallerIdentityResult>\n  <ResponseMetadata>\n    <RequestId>c56f5d68-8763-4032-a835-fd95efd83fa6</RequestId>\n  </ResponseMetadata>\n</GetCallerIdentityResponse>\n"
read 401 bytes
Conn keep-alive
I, [2020-03-17T15:47:45.275421 #9815]  INFO -- : [Aws::STS::Client 200 0.091573 0 retries] get_caller_identity()  

=> #<struct Aws::STS::Types::GetCallerIdentityResponse user_id="AIDAREALUSERID", account="5851234356", arn="arn:aws:iam::5851234356:user/test">

Note that:

  1. The set of headers sent by the AWS SDK differs from the set of headers that the MONGODB-AWS auth mechanism specification mentions. I used the AWS SDK implementation as a guide to determine the correct shape of the request to STS and in particular the Authorization header. The source code of Amazon's implementation is here and it generates, in particular, the x-amz-content-sha256` header which the MONGODB-AWS auth mechanism specification does not mention.
  2. This is a working request which can be replayed, making it possible to send this request that was created by the AWS SDK repeatedly with minor alterations to study STS error reporting behavior. STS as of this writing allows a 15 minute window during which a request may be replayed.
  3. The printed request only shows the headers and not the request body. In case of the GetCallerIdentity, the payload is fixed and is the same as what the MONGODB-AWS auth mechanism specification requires (Action=GetCallerIdentity&Version=2011-06-15).

Because the AWS SDK includes a different set of headers in its requests, it not feasible to compare the canonical requests generated by AWS SDK verbatim to the canonical requests generated by the driver.

Manual Requests

It is possible to manually send requests to STS using OpenSSL s_client tool in combination with the printf utility to transform the newline escapes. A sample command replaying the request printed above is as follows:

(printf "POST / HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded; charset=utf-8\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby3/3.91.1 ruby/2.7.0 x86_64-linux aws-sdk-core/3.91.1\r\nHost: sts.amazonaws.com\r\nX-Amz-Date: 20200317T194745Z\r\nX-Amz-Content-Sha256: ab821ae955788b0e33ebd34c208442ccfc2d406e2edc5e7a39bd6458fbb4f843\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAREALKEY/20200317/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=6cd3a60a2d7dfba0dcd17f9c4c42d0186de5830cf99545332253a327bba14131\r\nContent-Length: 43\r\nAccept: */*\r\n\r\n" &&
  echo "Action=GetCallerIdentity&Version=2011-06-15" &&
  sleep 5) |openssl s_client -connect sts.amazonaws.com:443

Note the sleep call - s_client does not wait for the remote end to provide a response before exiting, thus the sleep on the input side allows 5 seconds for STS to process the request and respond.

For reference, Amazon provides GetCallerIdentity API documentation .

Integration Test - Signature Generation

The Ruby driver includes an integration test for signature generation, where the driver makes the call to GetCallerIdentity STS endpoint using the provided AWS credentials. This test is in spec/integration/aws_auth_request_spec.rb.

STS Error Responses

The error responses produced by STS sometimes do not clearly indicate the problem. Below are some of the puzzling responses I encountered:

  • Request is missing Authentication Token: request is missing the Authorization header, or the value of the header does not begin with AWS4-. For example, this error is produced if the signature algorithm is erroneously given as AWS-HMAC-SHA256 instead of AWS4-HMAC-SHA256 with the remainder of the header value being correctly constructed. This error is also produced if the value of the header erroneously includes the name of the header (i.e. the header name is specified twice in the header line) but the value is otherwise completely valid. This error has no relation to the "session token" or "security token" as used with temporary AWS credentials.
  • The security token included in the request is invalid: this error can be produced in several circumstances:
    • When the AWS access key id, as specified in the scope part of the Authorization header, is not a valid access key id. In the case of non-temporary credentials being used for authentication, the error refers to a "security token" but the authentication process does not actually use a security token as this term is used in the AWS documentation describing temporary credentials.
    • When using temporary credentials and the security token is not provided in the STS request at all (x-amz-security-token header).
  • Signature expired: 20200317T000000Z is now earlier than 20200317T222541Z (20200317T224041Z - 15 min.): This error happens when x-amz-date header value is the formatted date (YYYYMMDD) rather than the ISO8601 formatted time (YYYYMMDDTHHMMSSZ). Note that the string 20200317T000000Z is never explicitly provided in the request - it is derived by AWS from the provided header x-amz-date: 20200317.
  • The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details: this is the error produced when the signature is not calculated correctly but everything else in the request is valid. If a different error is produced, most likely the problem is in something other than signature calculation.
  • The security token included in the request is expired: this error is produced when temporary credentials are used and the credentials have expired.

See also AWS documentation for STS error messages.

Resources

Generally I found Amazon's own documentation to be the best for implementing the signature calculation. The following documents should be read in order:

Signature Debugger

The most excellent awssignature.com was indispensable in debugging the actual signature calculation process.

MongoDB Server

MongoDB server internally defines the set of headers that it is prepared to handle when it is processing AWS authentication. Headers that are not part of that set cause the server to reject driver's payloads.

The error reporting when additional headers are provided and when the correct set of headers is provided but the headers are not ordered lexicographically can be misleading.

Direct AWS Requests

STS GetCallerIdentity API docs

When making direct requests to AWS, adding Accept: application/json header will return the results in the JSON format, including the errors.

AWS CLI

Configuration reference

Note that AWS CLI uses AWS_DEFAULT_REGION environment variable to configure the region used for operations.

AWS Ruby SDK

Configuration reference

Note that AWS Ruby SDK uses AWS_REGION environment variable to configure the region used for operations.

STS::Client#assume_role documentation

IMDSv2

X-aws-ec2-metadata-token-ttl-seconds is a required header when using IMDSv2 EC2 instance metadata requests. This header is used in the examples on Amazon's page describing IMDSv2, but is not explicitly stated as being required.

Not providing this header fails the PUT requests with HTTP code 400.

IAM Roles For EC2 Instances

Metadata Rate Limit

Amazon documentation states that the EC2 instance metadata endpoint is rate limited. Since the driver accesses it to obtain credentials whenever a connection is established, rate limits may adversely affect the driver's ability to establish connections.

Instance Profile Assignment

It can take over 5 seconds for an instance to see its instance profile change reflected in the instance metadata. Evergreen test runs seem to experience this delay to a significantly larger extent than testing in a standalone AWS account.

IAM Roles For ECS Tasks

ECS Task Roles

When an ECS task (or more precisely, the task definition) is created, it is possible to specify an execution role and a task role. The two are completely separate; an execution role is required to, for example, be able to send container logs to CloudWatch if the container is running in Fargate, and a task role is required for AWS authentication purposes.

The ECS task role is also separate from EC2 instance role and the IAM role for a user to assume a role - these roles all require different configuration.

AWS_CONTAINER_CREDENTIALS_RELATIVE_URI Scope

As stated in this Amazon support document, the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is only available to the PID 1 process in the container. Other processes need to extract it from PID 1's environment:

strings /proc/1/environment

Other ECS Metadata

strings /proc/1/environment also shows a number of other enviroment variables available in the container with metadata. For example a test container yields:

HOSTNAME=f893c90ec4bd
ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/5fb0b11b-c4c8-4cdb-b68b-edf70b3f4937
AWS_DEFAULT_REGION=us-east-2
AWS_EXECUTION_ENV=AWS_ECS_FARGATE
AWS_REGION=us-east-2
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/f17b5770-9a0d-498c-8d26-eea69f8d0924

Metadata Rate Limit

Amazon documentation states that ECS task metadata endpoint is subject to rate limiting, which is configured via ECS_TASK_METADATA_RPS_LIMIT container agent parameter. When the rate limit is reached, requests fail with 429 Too Many Requests HTTP status code.

Since the driver accesses this endpoint to obtain credentials whenever a connection is established, rate limits may adversely affect the driver's ability to establish connections.