Vault should use normal AWS credential chain to authenticate with AWS for AWS backend

[jszwedko]

It would be nice if vault could just use, e.g., an IAM instance role to authenticate itself with AWS for the AWS secret backend. This would alleviate the need to inject AWS credentials into vault (and these would automatically roll).

I was thinking we could just let github.com/awslabs/aws-sdk-go/aws do it's thing if the credentials are missing.

[armon]

@jszwedko I prefer having an explicit AWS key provided to Vault instead of using the instance IAM role. The primary reason is that the behavior is different depending on which Vault instance is active in an HA deploy scenario. With an explicit key, all Vaults would use the same credentials.

[jszwedko]

@armon I'm not saying we should remove the ability to set static credentials in the vault itself, but it would be nice to have the flexibility of using IAM roles. The issue you describe when running vault in HA is no different than running multiple of any other application that uses instance roles to talk to AWS -- it's on the operator to make sure that all of the instances running the application have the same role.

I'd be happy to work on a PR for this.

[jszwedko]

This also alleviates the need to create a separate process to roll the AWS credentials on a regular basis.

Let me know what you think!

[armon]

@jszwedko I see your point. If we pull this in, let's make it explicit that instance IAM roles are enabled so that it is maximally clear to operators where it is coming from. I'd be happy to review a PR with this functionality!

[jszwedko]

@armon awesome, I'll take shot!

Do you mean explicit in that it should still require a write to aws/<something> to enable usage of IAM roles? Or just a log message if it falls back to using IAM role?

Do you think we should also support setting via environment (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) or the file ~/.aws/credentials?

[sethvargo]

Hi @jszwedko

Do you mean explicit in that it should still require a write to aws/ to enable usage of IAM roles? Or just a log message if it falls back to using IAM role?

I think it should be a configuration for now. Maybe we can default it to true in a future release, but false should default for now.

Do you think we should also support setting via environment (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) or the file ~/.aws/credentials?

I think we should accept it as an argument for now. If you have the envvars set, it's as simple as $AWS_ACCESS_KEY_ID and if you have the credentials file set, it's as simple as a cat | grep | cut. I don't think it's worth doing in an initial pass, but definitely something to consider for a future enhancement.

What do you think?

I also would like to note that we just spent some time update the aws-sdk-go library and removing all deps on go-amz. So as of yesterday, Vault is 100% up-to-date with the library and future upgrades should be less painful.

[jszwedko]

Hey @sethvargo

I agree regarding making it a configuration option, this will make the behavior explicit.

I'll try to take a shot at this soon. We'd put secrets handling on the back burner for a little, but it is something we'll be addressing internally in the near future and using IAM with Vault would be a clear benefit for us.

Thanks!

[vaijab]

This does sound very scary.

[ajohnstone]

+1. Look forward to seeing these patches... @jszwedko

[ajohnstone]

1. HA is not supported for S3 backend storage. "s3 - Store data within an S3 bucket S3. This backend does not support HA." - https://vaultproject.io/docs/config/index.html
2. When configuring a cluster you typically use a autoscaling group, as such the instance profile attached to the launch configuration will dictate what IAM policy is applicable.
   a) Granted you can get skew with the policies when the instances are launched and you provide a new launch config. That is really down to the operator to ensure its correct

The typical case is you’re usually autoscaling in theses cases, but if you’re not, mis-configuring your cluster by picking the wrong IAM role for some machines is no different than mis-configuring your firewall when setting up some nodes in a cluster. @mpontes

With regards to shared credentials, not sure I see the point in supporting. Firstly IAM instance roles are more appropriate. Shared credentials ("~/.aws/credentials") would defeat the purpose of using vault to authenticate to the aws backend.

[jjshoe]

Behavior here should match how it works in terraform imho.

[Ginja]

I know it's bad/annoying to +1, but +1... :) Looking forward to seeing this feature.

[jefferai]

This is how auth is performed (autocorrect, doh) in the AWS credential backend. We're pretty swamped but if someone wanted to PR a change based on that it would be great!

[c4milo]

Deleted my previous message by mistake. I confirm that Vault is getting credentials from the AWS metadata service correctly. I changed my IAM role policy to refer to the resource as arn:aws:s3:::my_bucket instead of arn:aws:s3:::my_bucket/* and everything worked as expected.

[jefferai]

@jszwedko @ajohnstone @Ginja @jjshoe There's a branch at https://github.com/hashicorp/vault/tree/aws-cred-chain -- anyone that would like to help testing, it'd be much appreciated.

@c4milo S3 physical backend behavior is different than the AWS secret backend behavior, however, in that branch the S3 backend uses the new code too, so if you don't mind testing it'd be nice!