AWS provider config inconsistencies

[zoltrain]

Hello,

So I've been using Pulumi for the last 3 months, so firstly thanks for this awesome tool. I've found it really productive in comparison to Terraform.

Though I've just found a weird behaviour with "how' AWS configuration is loaded when running pulumi up.

I first had a look to see if anyone else has had a similar issue, and I found this closed Github issue, #252 so there seems to be a history of problems with the AWS provider specifically. Though I couldn't see any details in this thread on my particular issue so I think this might be "new".

Let's start with what I'm trying to achieve.

We have a primary account that has "sub" accounts under it's control, this is where things like IAM roles etc are managed from.

Now when I use the CLI I can populate my shell with the correct AWS env vars and target a profile that uses a different role in a different account without a hitch.

So this type of thing
~/.aws/credentials

[default]
aws_access_key_id =redacted
aws_secret_access_key = redacted

~/.aws/config

[profile my-profile]
role_arn = arn:aws:iam::{account id}:role/{role name}
source_profile=default

So if I do aws --profile my-profile iam list-roles I get the roles in the account bound to the my-profile profile.

Now when using the above AWS via Pulumi I get some inconsistent results "depending" on how I load up the config.

E.g. If I setup a provider like this

const myProvider = new aws.Provider('MyProvider', {
  assumeRole: {
    roleArn: 'same role ARN configured in my ~/aws/config file for my-profile`
  }
});

If I run a pulumi up the resources are created in the correct accounts.

Now if I configure the provider in code like yay

const myProvider = new aws.Provider('MyProvider', {
  profile: 'my-profile'
});

and run a pulumi up the resources are also created in the correct accounts.

but this is where things get weird, if you configure the above provider with the profile argument https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/aws/#ProviderArgs-profile AND you populate your current shell with AWS env vars from these credentials

[default]
aws_access_key_id =redacted
aws_secret_access_key = redacted

So your shell environment look like this

AWS_ACCESS_KEY_ID=redacted
AWS_SECRET_ACCESS_KEY=redacted

And run the same pulumi up it creates any resources configured against the custom myProvider against the default credentials. The role is never assumed.

Now why this isn't completely broken it's pretty non obvious behaviour, especially if a lot of your team rely on tools like https://github.com/remind101/assume-role to manage the different AWS profiles since these always export these environment variables.

Details of my Pulumi environment

❯ pulumi version
v2.0.0

❯ pulumi plugin ls
NAME  KIND      VERSION  SIZE    INSTALLED   LAST USED
aws   resource  2.1.0    223 MB  1 hour ago  1 hour ago

TOTAL plugin cache size: 223 MB

I've been able to reproduce this consistently with the above versions.

[leezen]

This behavior is documented in that if you set environment variables, they'll be preferred over anything else: https://www.pulumi.com/docs/intro/cloud-providers/aws/setup/#environment-variables

https://github.com/remind101/assume-role does the right thing with respect to this behavior because it sets all the appropriate environment variables, which includes the session token.

Closing for now, but please re-open if you observe anything that diverges from the above.

[zoltrain]

So in this case it might be worth documenting how the effects the underlining providers as well. I don't think it's an uncommon thing to want to target another role in another account via a provider in code. But I'd expect by declaring a provider saying "you must be this role" and magically bypassing seems pretty dangerous. Bearing in mind the underlying profile could be a completely different set of credentials. I'd either want a hard failure telling me this can have unexpected outcomes, or for it to always use the role I declare(whether it be by profile name or roleArn)

The reasoning is when you shift a set of code from say your laptop (configured with AWS creds) to CI (likely env vars set) the behaviour can have considerably different outcomes.

[zoltrain]

This should be possible to validate that the calling identity is in fact a the role you expect on the provider.

STS offers this API https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html

[leezen]

A couple thoughts:

1. Since it's all real code, you can totally just do what you described :) -- call GetCallerIdentity (which is exposed from pulumi-aws as well):

aws.getCallerIdentity({ provider: awsProvider}).then(identity => {
    // whatever checks you want to do
});

1. I've created a new issue to track what you're describing since it would very much make sense as you said to have a way to more easily say, "make sure it's this role" before continuing with execution of the program. Be able to declare a provider explicitly requiring a specific role #954

[zoltrain]

@leezen Good point, I can just throw if it doesn't match up to the role I'm expecting to be running "part" of the code as. But yeah having the providers pick this up, or documenting the fact that their config can be overridden by the env vars would be really helpful.