Incorrect AWS ID is used in ECS AWS auth

[nocode99]

Vault Version: 0.7.3

In ECS, we assign TaskRoleARN's to our ECS services/tasks.

$ vault write auth/aws/role/ecs-test \
    bound_iam_principal_arn=arn:aws:iam:1234567890:role/ecs-test \
    policies=ecs-test

$ vault read auth/aws/role/ecs-test
Key                             Value
---                             -----
allow_instance_migration        false
auth_type                       iam
bound_account_id              
bound_ami_id                  
bound_iam_instance_profile_arn
bound_iam_principal_arn         arn:aws:iam::1234567890:role/ecs-test
bound_iam_principal_id          ABCDEFGHIJKLMNOP
bound_iam_role_arn            
bound_region                  
bound_subnet_id               
bound_vpc_id                  
disallow_reauthentication       false
inferred_aws_region           
inferred_entity_type          
max_ttl                         0
period                          0
policies                        [default ecs-test]
resolve_aws_unique_ids          true
role_tag                      
ttl                             0

I then launch an ECS service and from there, I run:

$ vault auth -method=aws role=ecs-test
Error making API request.

URL: PUT https://vault.rocks:8200/v1/auth/aws/login
Code: 400. Errors:

* expected IAM assumed-role ec2_instance_profile_name to resolve to unique
AWS ID "ABCDEFGHIJKLMNOP" but got "PONMLKJIHGFEDCBA" instead

The invalid AWS ID is of the EC2 instance host of the container and not the container itself.

Is this something that would be resolved in the 0.7.4 release?

[joelthompson]

The vault CLI just uses whatever credentials the AWS Go SDK can pick up if they aren't specified explicitly on the command line, so it looks like the SDK is picking up the host's IAM instance profile credentials rather than the credentials supplied by the ECS task. Can you also run aws sts get-caller-identity from inside the ECS task?

Looking at http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html it mentions that you must be using at least version 1.11.0 of the ECS agent to pass in credentials. Can you confirm what version of the agent you're running? (The version of the go SDK used should definitely support pulling the task credentials.)

[nocode99]

Hi @joelthompson - yes I ran aws sts get-caller-identity on the ECS task and can confirm the AWS ID matches with the bound_iam_principal_id that gets added to the AWS role. I'm running the latest ECS agent 1.14.3 and have all the network settings on the host to use IAM Task Roles as per the doc you linked. We do a lot of batch processing and limit access to S3 buckets on the ECS task level and have been doing that with no issues.

I saw some other issues opened where iam:GetRole was required to the EC2 instance profile. Do I need to add the TaskRole IAM policy as well?

[joelthompson]

Do you have the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable set?

Looking at the SDK code, it looks like Go prioritizes AWS_CONTAINER_CREDENTIALS_FULL_URI variable, while the python code (which the aws cli uses) prioritizes AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, which could explain why the python and go SDKs are giving different results.

[nocode99]

No, it appears I have AWS_CONTAINER_CREDENTIALS_RELATIVE_URI set instead automatically by ECS. The value appears to be /v2/credentials/{{ container_hash_id }}. I tried unsetting it and trying to set AWS_CONTAINER_CREDENTIALS_FULL_URI instead but wasn't sure what it should be set to. I tried configuring it, but got the same results.

[jefferai]

@joelthompson is the auth CLI helper using the credsutil helper? That forms a credential chain in a specific order, so if there are creds in different places maybe it's not the order expected?

[joelthompson]

@jefferai -- aha! Yes, that's it. It needs to have a CredentialsEndpointProvider in the chain, which it doesn't.

Honestly, I think that just using AWS's defaults.RemoteCredProvider instead of generating our own instance of ec2rolecreds.EC2RoleProvider would be the simplest thing, as it checks for the necessary environment variables, generates the magic URLs necessary, etc., and will just fall back to ec2rolecreds.EC2RoleProvider if those environment variables aren't set. It's also more future proof, in case AWS adds yet another way to get credentials.

The only real difference I see is defaults.RemoteCredProvider defaults to an ExpiryWindow of 5*time.Minute while the awsutils.GenerateCredentialChain has an ExpiryWindow of 15, i.e., 15ns, which doesn't seem like a meaningful value, I'm guessing 15*time.Minute was intended, and we could even change that back on the returned config provider.

Thoughts on just moving to the RemoteCredProvider? I'll be happy to put together a PR for this.

[jefferai]

See #2983

[nocode99]

So I was just using the CLI in order to test to see if 0.7.3 issue resolved our ECS issues. I'm currently trying to do this via the API, but however I encode the request_headers, it doesn't seem Vault likes the decoded format. I've tried a number of different formats, but still seem to be having trouble. Can you post an example format of the iam_request_headers before it's base64 encoded?

[jefferai]

@nocode99 Each header key maps to an array of values. so it'd be something like { "x-aws-something": ["myvalue"], "x-aws-somethingelse": ["myothervalue"]}

[joelthompson]

@nocode99 -- something like this:

{
    "Authorization": [
        "AWS4-HMAC-SHA256 Credential=AXXXXXXXX/20170707/us-east-1/sts/aws4_request, SignedHeaders=host;x-amz-date;x-amz-security-token;x-vault-awsiam-server-id, Signature=deadbeef"
    ],
    "X-Amz-Date": [
        "20170707T162309Z"
    ],
    "X-Amz-Security-Token": [
        "XXXXXXXX"
    ],
    "X-Vault-awsiam-Server-Id": [
        "vault.example.com"
    ]
}

If you're using python, I have a gist for how to generate the values in python2 and python3

[nocode99]

@joelthompson thanks! That worked from ECS container. We are primarily a python shop so this will definitely come in handy.

[joelthompson]

Awesome, glad I could help!

[jefferai]

Should be fixed by #2983 being merged!

[jasonmf]

I encountered this error with a different cause. Documenting for future Vaultoviles.

We were getting this error referring to a seemingly arbitrary role. The role turned out to be the role of the in-VPC HTTP proxy. The proxy wasn't blocking 169.254.169.254 and the client wasn't ignoring the proxy settings for 169.254.169.254. The former is itself a security issue but either one of these being set correctly would have made the issue disappear or be obviousl