Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot authenticate on AWS using switchrole #252

Closed
jnancel opened this issue Jun 29, 2018 · 26 comments · Fixed by #580 or #874
Closed

Cannot authenticate on AWS using switchrole #252

jnancel opened this issue Jun 29, 2018 · 26 comments · Fixed by #580 or #874
Assignees
Labels
area/providers customer/feedback Feedback from customers kind/bug Some behavior is incorrect or out of spec p1 Bugs severe enough to be the next item assigned to an engineer
Milestone

Comments

@jnancel
Copy link

jnancel commented Jun 29, 2018

Hello,

I'm trying to bootstrap my first pulumi project but I'm encountering an issue with the supported authentication method on AWS.

If I'm using a profile in which my credentials are defined ( with aws_access_key_id and aws_secret_access_key ), it works like a charm. But that's not how we work in my company, we have a source account and we're using switchrole to jump from that source account to all other accouts ( this is standard AWS procedure when you're dealing with multiple accounts ).

Here's the look of my profiles in ~/.aws/config :

[profile myprofile]
role_arn = arn:aws:iam::<AWS destination account id>:role/<the role I'm going to impersonate>
mfa_serial = arn:aws:iam::<source AWS account ID>:mfa/<IAM User name on the source account>
source_profile = <the profile name in which my keys are defined for my source account>

Now when I'm using one of those profile, it fails :

$ AWS_PROFILE=myprofile pulumi update
Previewing update of stack 'prod'
Previewing changes:

     Type                    Name               Plan          Info
 *   global                  global             no change     1 error
 +   └─ pulumi:pulumi:Stack  first-pulumi-prod  create        
 
Diagnostics:
  global: global
    error: unable to discover AWS AccessKeyID and/or SecretAccessKey - see https://pulumi.io/install/aws.html for details on configuration
 
error: an error occurred while advancing the preview

When using the same command line with my source profile ( the one with the keys ), I have the expected result :

      Type                 Name               Plan       Info
 +   pulumi:pulumi:Stack  first-pulumi-prod  create
 +   └─ aws:s3:Bucket     my-bucket          create

info: 2 changes previewed:
    + 2 resources to create

Do you want to perform this update?
  yes
> no
  details

Is there any configuration I'm missing or is it not possible for the moment to use an AWS profile with switchrole ?

Thanks

@ellismg
Copy link
Contributor

ellismg commented Jun 29, 2018

Can you try setting AWS_SDK_LOAD_CONFIG=1 in addition to setting AWS_PROFILE? The underlying provider uses the AWS GO SDK and it does not read ~/.aws/config by default. I would be interested in knowing it that works.

If it does, I wonder if this is just something we should be setting ourselves in some cases, as this is surprising behavior.

@jnancel
Copy link
Author

jnancel commented Jul 2, 2018

Hello Matt,

Thanks for the answer, I tried enabling that option but encountered a panic error :

$ AWS_PROFILE=myprofile pulumi preview
Previewing update of stack 'prod'
Previewing changes:

     Type    Name    Plan          Info
 *   global  global  no change     1 error, 2 info messages
 
Diagnostics:
  global: global
    info: 2018/07/02 09:45:04 ERROR: failed to create session with AWS_SDK_LOAD_CONFIG enabled. Use session.NewSession to handle errors occurring during session creation. Error: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.
 
  global: global
    info: panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x8f3b4e]
    goroutine 34 [running]:
    github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata.unmarshalHandler(0xc4206c0800)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata/service.go:119 +0x3e
    github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc4206c09e8, 0xc4206c0800)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/request/handlers.go:213 +0x9d
    github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc4206c0800, 0x0, 0x0)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:525 +0x4d9
    github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata.(*EC2Metadata).GetMetadata(0xc42000e780, 0x2e2dfc4, 0xb, 0x0, 0x2e2dcc2, 0xb, 0x1)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata/api.go:28 +0x18b
    github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata.(*EC2Metadata).Available(0xc42000e780, 0xc420495a00)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata/api.go:129 +0x40
    github.com/pulumi/pulumi-aws/vendor/github.com/terraform-providers/terraform-provider-aws/aws.GetCredentials(0xc4200a0280, 0x0, 0x0, 0x0)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/terraform-providers/terraform-provider-aws/aws/auth_helpers.go:164 +0x1ba5
    github.com/pulumi/pulumi-aws.preConfigureCallback(0xc420290870, 0xc420734000, 0xc420208db0, 0xc420208e40)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/resources.go:166 +0x411
    github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi-terraform/pkg/tfbridge.(*Provider).Configure(0xc420239770, 0x4acab20, 0xc4202907e0, 0xc42000e060, 0xc420239770, 0x1, 0x1)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi-terraform/pkg/tfbridge/provider.go:227 +0xdf7
    github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Configure_Handler.func1(0x4acab20, 0xc4202907e0, 0x2a72c20, 0xc42000e060, 0x4acab20, 0xc4202907e0, 0x4acfda0, 0x4b68c68)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go/provider.pb.go:699 +0x86
    github.com/pulumi/pulumi-aws/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x4acab20, 0xc4202907e0, 0x2a72c20, 0xc42000e060, 0xc4205342a0, 0xc4205342c0, 0x0, 0x0, 0x4abbca0, 0xc4202532e0)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:61 +0x326
    github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Configure_Handler(0x2d73640, 0xc420239770, 0x4acab20, 0xc4205942a0, 0xc42030e410, 0xc4204ce560, 0x0, 0x0, 0x30, 0x2aeb560)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go/provider.pb.go:701 +0x16d
    github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc4200b2000, 0x4ace9a0, 0xc4200b0000, 0xc420572200, 0xc42026dbc0, 0x4b3d460, 0x0, 0x0, 0x0)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc/server.go:826 +0xab4
    github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4200b2000, 0x4ace9a0, 0xc4200b0000, 0xc420572200, 0x0)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc/server.go:1023 +0x1528
    github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4202ad320, 0xc4200b2000, 0x4ace9a0, 0xc4200b0000, 0xc420572200)
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc/server.go:572 +0x9f
    created by github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    	/home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc/server.go:570 +0xa1
 
  global: global
    error: transport is closing
 
error: an error occurred while advancing the preview

That's when using the profile in my ~/.aws/config with switchrole. When using the profile in ~/.aws/credentials with the keys, I got the expected behavior.

Anyway, in my opinion, I think it would be best to check the content of both files by default.

Thanks

@joeduffy joeduffy added kind/bug Some behavior is incorrect or out of spec area/providers labels Sep 4, 2018
@joeduffy joeduffy added this to the 0.18 milestone Sep 4, 2018
@joeduffy joeduffy assigned jen20 and unassigned pgavlin Sep 24, 2018
@joeduffy
Copy link
Member

@jen20 just offered to help track this down, so reassigning.

@lukehoban lukehoban modified the milestones: 0.18, 0.19 Oct 19, 2018
@jen20
Copy link
Contributor

jen20 commented Nov 26, 2018

Hi @jnancel! Sorry for the delay in replying here. I committed an example of using AssumeRole to the pulumi/examples repository a few weeks back - it's here. Can you give that approach a try?

I'll close out this issue for now since there is a way to do this - please feel free to comment further if you think this doesn't address your requirements though and we'll take another look!

@jen20 jen20 closed this as completed Nov 26, 2018
@lukehoban
Copy link
Member

@jen20 That example shows how to use an assumed role programmatically from within Pulumi.

But do we know why the steps @jnancel tried don' work? Is it expected that setting AWS_PROFILE will not work? Is it expected that also settings AWS_SDK_LOAD_CONFIG=1 causes a panic? Can pulumi config set aws:profile <profile> be used instead correctly? This approach of trying to set a profile from outside of the pulumi invocation feels like it will be common.

@lukehoban lukehoban reopened this Jan 5, 2019
@lukehoban
Copy link
Member

Re-opening. I hit this again recently myself, and was able to work around via an approach like the example, but as expected to just be able to set the profile externally.

@lukehoban lukehoban modified the milestones: 0.19, 0.20 Jan 5, 2019
@lukehoban lukehoban modified the milestones: 0.20, 0.21 Jan 16, 2019
@lukehoban
Copy link
Member

This has come up from users several times - I'd love to nail down any remaining issues here and document the expected modes that work.

@jen20 jen20 modified the milestones: 0.21, 0.22 Mar 7, 2019
@lukehoban lukehoban modified the milestones: 0.22, 0.23 Apr 23, 2019
@eshamay
Copy link

eshamay commented May 10, 2019

I've just run into a strikingly similar issue.

I have the following setup:

$ echo $AWS_PROFILE
online-sandbox-admin
$ cat ~/.aws/config
[profile online-sandbox-admin]
source_profile = bastion
region = us-west-2
role_arn = arn:aws:iam::redact:role/online-sandbox-admin-role
$ cat ~/.aws/credentials
[bastion]
aws_access_key_id = redact
aws_secret_access_key = redact
region = us-west-2
aws_session_token = redact
$ aws sts get-caller-identity
{
    "Account": "redact",
    "UserId": "redact-session-1557521500",
    "Arn": "arn:aws:sts::redact:assumed-role/online-sandbox-admin-role/redact-session-1557521500"
}

When using the aws CLI directly then things work exactly as expected (e.g. list roles, instances, etc). I can see all the resources in my account. However, running pulumi refresh lists everything as delete meaning that pulumi can't find the resources.

@lukehoban lukehoban added the customer/feedback Feedback from customers label May 10, 2019
@brandonbloom
Copy link

We use AssumeRole as well. I'd like to cast my vote for supporting the aws:profile configuration setting.

@keen99
Copy link

keen99 commented May 23, 2019

+1 for native role assumption for the CLI tools based on AWS_PROFILE, aws:profile, and other profile setting methods. it's a significant blocker to not support that for any kind of modern multi account setup.

eshamay's setup is pretty typical (and may also include an MFA component), and the aws cli tools handle this smoothly now, but pulumi just does:

    error: unable to discover AWS AccessKeyID and/or SecretAccessKey - see https://pulumi.io/install/aws.html for details on configuration

@stack72 stack72 modified the milestones: 0.30, 0.31 Dec 16, 2019
@lukehoban lukehoban assigned mikhailshilkov and unassigned stack72 Jan 22, 2020
@mikhailshilkov
Copy link
Member

Can anyone summarize the remaining work here for me, or is the issue that nobody knows?

Unfortunately, I'm a noob in the fascinating world of AWS IAM, so I may need some hand-holding here.

@lukehoban lukehoban modified the milestones: 0.31, 0.32 Feb 7, 2020
@cleverguy25
Copy link

I am seeing this issue as well, and I swore it worked once upon a time.

@lukehoban
Copy link
Member

@cleverguy25 which issue specifically are you hitting? (This GitHub issue has unfortunately tracked 3 different things in its lifetime).

@cleverguy25
Copy link

cleverguy25 commented Feb 11, 2020

Context, I am using WSL on windows. I am logged on to our bastian account, and doing the whole source profile thing to assume a role in another account. AWS cli works, but pulumi gives me this error:

Error: Invoke of 'aws:secretsmanager/getSecretVersion:getSecretVersion' failed: rpc error: code = Unknown desc = invocation of aws:secretsmanager/getSecretVersion:getSecretVersion returned an error: unable to discover AWS AccessKeyID and/or SecretAccessKey - see https://pulumi.io/install/aws.html for details on configuration ()

@lukehoban
Copy link
Member

With the setup below - things work.

In ~/.aws/credentials:

[pulumi-token]
aws_access_key_id = <redacted>
aws_secret_access_key = <redacted>
aws_session_token = <redacted>

In ~/.aws/config:

[profile staging-token]
role_arn=arn:aws:iam::<redacted>:role/OrganizationAccountAccessRole
source_profile=pulumi-token
region=us-west-2

In Pulumi.<stack>.yaml:

config:
  aws:profile: staging-token
  aws:region: us-west-2

Results:

$ pulumi up
Previewing update (staging):

     Type                 Name                Plan       
 +   pulumi:pulumi:Stack  assumerole-staging  create     
 +   └─ aws:s3:Bucket     my-bucket           create     
 
Resources:
    + 2 to create

Do you want to perform this update? yes
Updating (staging):

     Type                 Name                Status      
 +   pulumi:pulumi:Stack  assumerole-staging  created     
 +   └─ aws:s3:Bucket     my-bucket           created     
 
Outputs:
    bucketName: "my-bucket-77c7547"

Resources:
    + 2 created

Duration: 1m27s

However, if I remove the aws:profile setting and instead set the AWS_PROFILE env var I can reproduce:

  aws:s3:Bucket (my-bucket):
    error: unable to discover AWS AccessKeyID and/or SecretAccessKey - see https://pulumi.io/install/aws.html for details on configuration

I presume that provides a workaround for anyone hitting this (use pulumi config set aws:profile <profile>), though we should also fix the AWS_PROFILE env var support.

Are there any other configurations folks have that are also hitting this error?

lukehoban added a commit that referenced this issue Feb 11, 2020
For both programmatic access, and for pre configuration, we must read in all the appropriate env vars.

Fixes #252
stack72 pushed a commit that referenced this issue Feb 12, 2020
For both programmatic access, and for pre configuration, we must read in all the appropriate env vars.

Fixes #252
stack72 pushed a commit that referenced this issue Feb 12, 2020
For both programmatic access, and for pre configuration, we must read in all the appropriate env vars.

Fixes #252
lukehoban added a commit that referenced this issue Feb 12, 2020
For both programmatic access, and for pre configuration, we must read in all the appropriate env vars.

Fixes #252
lukehoban added a commit that referenced this issue Feb 12, 2020
For both programmatic access, and for pre configuration, we must read in all the appropriate env vars.

Fixes #252
@geof2001
Copy link

geof2001 commented Mar 31, 2021

New to pulumi and have tried having both aws:profile set in Pulumi.env.yaml and env set with AWS_PROFILE individually and set together and have run into the same error: unable to discover AWS AccessKeyID and/or SecretAccessKey - see https://pulumi.io/install/aws.html for details on configuration each time. This is pretty basic authentication using ootb AWS IAM methods. Hard to even get interested in this as a project without basic auth methods being nailed down properly. No issues with aws CLI with the env setup with AWS_PROFILE.

@lukehoban
Copy link
Member

@geof2001 would love anymore details you can share on your setup. The usage pattern you describe is definitely supported in general, and used by many/most Pulumi users on AWS - so I expect there is some other more subtle twist on the scenario in your environment. Without sharing any sensitive data - can you share details of how you have your profile configured?

@geof2001
Copy link

geof2001 commented Mar 31, 2021

~/.aws/credentials is populated with account profiles like this

[profile-admin]
aws_access_key_id=redact
aws_secret_access_key=redact
aws_session_token=redact

Have tried setting env var for export AWS_PROFILE=profile-admin to this as well as setting pulumi config set aws:profile profile-admin individually and separately and get the error posted in prior message.

The only way I've been able to get anything to work is to set all three ENV variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN

while this is feasible for a POC it is not very scalable. I would have expected the recommended settings of the ENV variable pulumi config setting to work fully on their own.

We use OneLogin to authenticate SSO and create sessions with temporary keys so the three tokens would have to be changed every time you setup a new session.

My environment is Ubuntu on WSL, have setup and used terraform with AWS profiles and can use profiles directly from python using boto3.setup_default_session calls.

If there is more detail you'd like please let me know and will try to specify

@infin8x infin8x added the p1 Bugs severe enough to be the next item assigned to an engineer label Jul 10, 2021
@rubiagatra
Copy link

is there any update for this?

@mati0090
Copy link

mati0090 commented Aug 4, 2021

I had same issue with AWS CLI configured. The reason was aws.Provider set for resources with non-default profile defined:

export const provider = new aws.Provider("infra", {
  profile: 'test-account',
  region: aws.config.requireRegion(),
});

Removing profile setting fixed an issue.

@yann-soubeyrand
Copy link

yann-soubeyrand commented May 11, 2022

[EDIT] Sorry for the noise, my issue seems to be related to pulumi/pulumi-eks#647.

Hello @lukehoban, I’m using this small example in go

package main

import (
	"github.com/pulumi/pulumi-eks/sdk/go/eks"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// Create an EKS cluster with the default configuration.
		cluster, err := eks.NewCluster(ctx, "my-cluster", nil)
		if err != nil {
			return err
		}

		// Export the cluster's kubeconfig.
		ctx.Export("kubeconfig", cluster.Kubeconfig)
		return nil
	})
}

and when I do a pulumi up, I get

Previewing update (test):
     Type                 Name                      Plan     Info
     pulumi:pulumi:Stack  pulumi-test-test           1 error; 10 messages
 
Diagnostics:
  pulumi:pulumi:Stack (pulumi-test-test):
    Error: It looks like you're using AWS profiles. Please specify this profile in providerCredentialOpts.: Error: It looks like you're using AWS profiles. Please specify this profile in providerCredentialOpts.
        at new Cluster (/var/home/yann/.pulumi/plugins/resource-eks-v0.37.1/node_modules/@pulumi/cluster.ts:1396:19)
        at Object.construct (/var/home/yann/.pulumi/plugins/resource-eks-v0.37.1/node_modules/@pulumi/cmd/provider/cluster.ts:21:29)
        at Provider.construct (/var/home/yann/.pulumi/plugins/resource-eks-v0.37.1/node_modules/@pulumi/cmd/provider/index.ts:123:24)
        at Server.<anonymous> (/var/home/yann/.pulumi/plugins/resource-eks-v0.37.1/node_modules/@pulumi/provider/server.ts:329:48)
        at Generator.next (<anonymous>)
        at fulfilled (/var/home/yann/.pulumi/plugins/resource-eks-v0.37.1/node_modules/@pulumi/pulumi/provider/server.js:18:58)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)
 
    error: program failed: waiting for RPCs: rpc error: code = Unknown desc = It looks like you're using AWS profiles. Please specify this profile in providerCredentialOpts.
    exit status 1
 
    error: an unhandled error occurred: program exited with non-zero exit code: 1

Additional informations

❯ pulumi version
v3.32.1

❯ pulumi plugin ls -p    
NAME        KIND      VERSION  SIZE    INSTALLED  LAST USED
aws         resource  4.15.0   276 MB  n/a        57 minutes ago
eks         resource  0.37.1   1.6 MB  n/a        24 minutes ago
go          language           22 MB   n/a        4 hours ago
kubernetes  resource  3.0.0    78 MB   n/a        57 minutes ago

TOTAL plugin cache size: 378 MB

❯ go version 
go version go1.18 linux/amd64

❯ cat go.mod
module pulumi-test

go 1.16

require (
	github.com/pulumi/pulumi-eks/sdk v0.37.1
	github.com/pulumi/pulumi/sdk/v3 v3.32.1
)

❯ cat ~/.aws/config                                                                            
[profile main]

[profile sandbox]
source_profile = main
role_arn = <arn>
region = eu-west-1

❯ cat ~/.aws/credentials 
[c2c]
aws_access_key_id = <key-id>
aws_secret_access_key = <secret>

❯ env | grep AWS              
AWS_PROFILE=sandbox

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/providers customer/feedback Feedback from customers kind/bug Some behavior is incorrect or out of spec p1 Bugs severe enough to be the next item assigned to an engineer
Projects
None yet