steam pipe fails to work when using aws sso credentials

[thapabishwa]

My organization recently made the switch from aws iam user to aws sso credentials(compliance reasons). As such, when using sso credentials steampipe fails to work.

~ >>> aws sso login --profile myprofile                                                                                                                           
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://device.sso.us-west-2.amazonaws.com/

Then enter the code:

SAMPLE-CODE
Opening in existing browser session.
Successully logged into Start URL: https://sampleurl.awsapps.com/start#/
~ >>> steampipe query                                                                                                                                                       
Welcome to Steampipe v0.2.2
For more information, type .help
> select * from aws_account
Error: pq: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
>

[LalitLab]

Thanks, @thapabishwa for trying out Steampipe

It is in our list of features for AWS Plugin and will try to support in the upcoming version of the https://github.com/turbot/steampipe-plugin-aws plugin

[thapabishwa]

Thanks for letting me know that it is in the roadmap @LalitTurbot. Let me know if I can help you out in anyways.

[e-gineer]

Hey @thapabishwa ... Under the hood the aws sso just creates a temporary credential set. These are often also available through the AWS SSO UI. AWS Docs.

So until we have direct support in steampipe, you can consider using steampipe's connection management to set those temporary credentials? Hopefully that will unblock you.

[thapabishwa]

Thanks for that @e-gineer . It would at-least unblock me for a while.

[cbruno10]

Hi @thapabishwa, this should be fixed in the latest version of the AWS plugin (v0.6.0). You may also need to update your AWS CLI v2 version to the latest one as we found some older versions had issues with the AWS Go SDK.

Can you please give it another try and let us know if using SSO creds with Steampipe works for you?

[iamjoeker]

I'm having the same issue with using aws-vault which after obtaining credentials from either Keychain or STS sets environment variables and runs a command or spawns a new shell.

My Steampipe aws plugin config is empty which per the docs says it should use the normal sdk resolution order looking first to the environment. aws CLI and other sdk tools consume these creds without issue. Steampipe returns the same as OP. I've not yet figured out how to set the referenced setting for verbose messaging.

[e-gineer]

@iamjoeker thanks for trying steampipe and reporting this! I’m not sure why we are not picking up those creds for you, we’ll take a look into this as well.

[LalitLab]

@iamjoeker
I have tried to explore a bit around it.

I am the latest version(0.6.0) of the Steampipe AWS plugin.
Please use steampipe plugin update aws to update to the latest version of the AWS plugin.

Here

$ aws-vault add lalit
Enter Access Key Id: ABDCDEFDASDASF
Enter Secret Key: %%%

$ aws-vault exec lalit -- env | grep AWS
AWS_DEFAULT_REGION=us-east-1
AWS_VAULT=vault1
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=dummy_access_id
AWS_SECRET_ACCESS_KEY=dummy_secret_key
AWS_SESSION_TOKEN=dummy_session_token
AWS_SECURITY_TOKEN=dummy_session_token
AWS_SESSION_EXPIRATION=2021-03-06T14:58:05Z

$ export AWS_ACCESS_KEY_ID=dummy_access_id
$ export AWS_SECRET_ACCESS_KEY=dummy_secret_key
$ export AWS_SESSION_TOKEN=dummy_session_token

 $ aws iam list-users
 
 An error occurred (InvalidClientTokenId) when calling the ListUsers operation: The security token included in the request is invalid
 $ aws s3 ls         
 2021-02-05 00:02:51 aws-test-bucket-123

$ steampipe query                        
Welcome to Steampipe v0.2.3
For more information, type .help

> select name from aws.aws_s3_bucket
+-------------------------------------------------+
| name                                            |
+-------------------------------------------------+
| aws-test-bucket-123                             |
+-------------------------------------------------+

> select * from aws.aws_iam_user
Error: InvalidClientTokenId: The security token included in the request is invalid
	status code: 403, request id: eb263f03-65cc-45d9-9e4d-bd408cd6ed25

Seems like it is working for tables which doesn't use AWS IAM service.

https://github.com/99designs/aws-vault/blob/master/USAGE.md#temporary-credentials-limitations-with-sts-iam

Please let us know if we should try a different way to replicate the problem.

[iamjoeker]

@LalitTurbot Thanks for investigating. Here's a capture of what I'm seeing. I tried with and without sessions. The capture below is not using sessions. I also tried dropping all AWS environment variables with the exception of the key id and secret with the same results.

Let me know if there is any additional info or testing I can do to help. If I have time next weekend, I'll pull source and try to investigate further from there.

 ~ > aws-vault exec -n my-profile
 ~ > env|grep AWS_
AWS_VAULT=my-profile
AWS_DEFAULT_REGION=us-east-2
AWS_REGION=us-east-2
AWS_ACCESS_KEY_ID=XXXXXX
AWS_SECRET_ACCESS_KEY=YYYYYYYYY
 ~ > PAGER= aws iam list-users
{
    "Users": [
        {
            "Path": "/",
            "UserName": "XXXXX",
            "UserId": "XXXXXXXXX",
            "Arn": "arn:aws:iam::999999999:user/XXXXXX",
            "CreateDate": "2018-07-17T23:49:14+00:00",
            "PasswordLastUsed": "2021-03-03T19:34:18+00:00"
        }
    ]
}
 ~ > PAGER= aws s3api list-buckets
{
    "Buckets": [
        {
            "Name": "config-bucket-999999999",
            "CreationDate": "2019-08-09T06:28:41+00:00"
        }
    ],
    "Owner": {
        "ID": "XXXXXXXXX"
    }
}
 ~ > steampipe --version
steampipe version 0.2.3

 ~ > steampipe plugin list
+--------------------------------------------------+---------+-------------+
| NAME                                             | VERSION | CONNECTIONS |
+--------------------------------------------------+---------+-------------+
| hub.steampipe.io/plugins/turbot/aws@latest       | 0.6.0   | aws         |
| hub.steampipe.io/plugins/turbot/steampipe@latest | 0.1.1   | steampipe   |
+--------------------------------------------------+---------+-------------+
 ~ > steampipe query
Welcome to Steampipe v0.2.3
For more information, type .help
> select * from aws.aws_iam_user;
Error: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
> select * from aws.aws_s3_bucket;
Error: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
>

[LalitLab]

@iamjoeker
can you please share details of ~/.steampipe/aws.spc

[iamjoeker]

@LalitTurbot Below:

connection "aws" {
  plugin    = "aws"
}

Although...Good news to report. I updated the aws plugin to 0.8.0 (0.6.0 was the latest as of my original post) and now everything is working as expected. Aws-vault with STS role assumption, session, and just access keys all working so far.

Thanks so much for the troubleshooting assistance!

[e-gineer]

Awesome time hear we're running smoothly with SSO and aws-vault in particular. Thanks @iamjoeker for confirming we've got this issue nailed 👍. We appreciate your help with the troubleshooting!

[jweyrich]

@e-gineer When I run a query on IAM on account prod using aws-vault, example:

aws-vault exec prod -- steampipe query "select name from aws_iam_user;"

It fails with:

Warning: executeQueries: query 1 of 1 failed: ERROR: rpc error: code = Unknown desc = InvalidClientTokenId: The security token included in the request is invalid
status code: 403, request id: (SQLSTATE HV000)

Querying other services (non-IAM) on the same account/profile seem to work fine. Example:

aws-vault exec prod -- steampipe query "select name from aws_s3_bucket;"

The s3 list works flawlessly.

Additionally, the profile prod is configured using an access key + secret key from an user with AdministratorAccess policy. And I have another AWS account configured on profile homolog which is accessed thru AssumeRole from the prod account/profile. The same IAM query works fine for homolog.

Here's my env environment:

macOS 10.15.7
aws-vault v6.6.0
steampipe version 0.15.0
plugins:
| hub.steampipe.io/plugins/turbot/aws@latest | 0.66.0 | aws |

Not sure what it is.

[cbruno10]

@jweyrich I've created a new issue for you in turbot/steampipe-plugin-aws#1127, as this is a better place to discuss AWS plugin issues, and I don't necessarily want to re-open this thread as the original issue was different than what you're encountering.