SSO session expiration and re-login

[villelahdenvuo]

Describe the bug

I'm using AWS SSO to authenticate and aws-sdk-js-v3. When the SSO session expires it tells me to log in again with an error message, but even after repeating the aws sso login, the credentials still don't work.

Expected Behavior

The Node server is able to load credentials again and I can keep developing.

Current Behavior

The error message persists.

Reproduction Steps

1. I log in normally using aws sso login --profile developer
2. I start my Node.js server: AWS_PROFILE=developer node server.js
3. I keep developing my app until my SSO session expires
4. aws-sdk-js-v3 throws an error:

CredentialsProviderError: The SSO session associated with this profile has expired. To refresh this SSO session run aws sso login with the corresponding profile

5. I login again aws sso login --profile developer
6. Same error is thrown again when I try to call some aws-sdk-js-v3 command.

Possible Solution

No response

Additional Information/Context

I opened my .aws/sso/cache/xxx.json file and notice that the accessToken is updated however the expiresAt property is not updated even after a new aws sso login call.

If I wait more time and run aws sso login again it finally updates the expiresAt property.

It seems that this is a likely reason for the issue I'm having. It seems like there is some mismatch with different expirations.

CLI version used

aws-cli/2.8.2 Python/3.10.7 Darwin/21.6.0 source/arm64 prompt/off

Environment details (OS name and version, etc.)

MacOS

[tim-finnigan]

Hi @villelahdenvuo thanks for reaching out. Per this documentation: "Each time a user signs in to IAM Identity Center, a sign in session is created with an 8-hour lifetime."

Regarding what you said here:

If I wait more time and run aws sso login again it finally updates the expiresAt property.

How much more time do you have to wait in this scenarios?

[michelleparent]

I'm also seeing the same issue as @villelahdenvuo. accessToken gets updated, but expiresAt does not after running aws sso login --profile <>. A second run of the command updated the expiresAt property. This was tested within a 10 minute window.

[tim-finnigan]

@michelleparent which version of the AWS CLI are you using? There was a recent issue involving SSO and it looks like the issue creator was also using v2.8.2 like @villelahdenvuo.

I recommend updating to the latest version (currently 2.8.6 per the CHANGELOG) and let us know if you're still seeing the issue.

[michelleparent]

@tim-finnigan I upgrades to v2.8.7 and I'm still seeing the issue. I tried clearing my SSO cache in ~/.aws/sso/cache, but that didn't help anything.

[villelahdenvuo]

How much more time do you have to wait in this scenarios?

@tim-finnigan I would say waiting around 5-10 minutes was enough, but it's a very annoying disturbance to development because I cannot use my development environment during that time.

I will update to the latest version and keep trying, but like you said it only happens after 8 hours so it's not something that happens every day.

[villelahdenvuo]

You're right about the cause. It's the session that remains active on the portal, but somehow the generated tokens are already expired a bit sooner. If I clear the cookies on the portal page I do get a new expiresAt value, so that can potentially help as a workaround, but still an annoying disturbance to experience.

[tim-finnigan]

@michelleparent are you still experiencing this issue? If so could you add share your debug logs with sensitive info redacted by adding --debug to the command?

Thanks @villelahdenvuo for the feedback, I can forward your comments to the SSO team - please let me know if there's anything you'd like to expand on. You could also consider reaching out through AWS Support for further assistance.

[spg]

I had the same issue. I confirm that clearing the cookies for https://{your-org-id}.awsapps.com works.

Version info:

➜  aws --version                           
aws-cli/2.6.1 Python/3.9.11 Darwin/21.5.0 exe/x86_64 prompt/off

[tim-finnigan]

Based on the recent comments I think this issue can now be closed. If there's any other feedback to share regarding this please let us know. Thanks!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[villelahdenvuo]

@tim-finnigan I wouldn't say this issue can be closed, perhaps it needs to be moved to another project? The problem still persists, clearing the cookies is not a good workaround since it still breaks the development workflow. The tokens should just work or the CLI should tell me to sign in again, not tell me token is expired and provide me no way to refresh it (until I wait or clear cookies manually).

[tim-finnigan]

@villelahdenvuo thanks for following up. I can transfer this to our cross-SDK repository to track as a feature request for the SSO team. If you could summarize the request then I'll do that and forward your feedback to that team.

[villelahdenvuo]

@tim-finnigan It's difficult to summarize concisely, but here's an attempt:

Ensure that AWS SDK and AWS CLI token expiration & refresh logic work together properly with an AWS SSO session. Currently SDK token can expire while the SSO session is still valid causing a problem where SDK says expired and CLI says you're good to go when you try to do a aws sso login to refresh your expired token causing the token not to be updated.

[HeskethGD]

We experienced this problem too. Having to clear cookies works but is not a practical solution. Hopefully this gets looked at.

[tim-finnigan]

Thanks @villelahdenvuo for following up. I'm now wondering if what you're describing overlaps with other previous issues such as aws/aws-cli#5955. Can you describe your SSO configuration and confirm? The guidance in this comment specifically may be helpful. Please let us know.

[villelahdenvuo]

@tim-finnigan No, we're using a barebones SSO setup for example:

[profile shared-developer]
sso_account_id = xxx
sso_role_name = DeveloperAccess
sso_start_url = https://yyy.awsapps.com/start/
sso_region = eu-west-1
region = eu-west-1
output = json

With @aws-sdk/credential-provider-node.

[ppena-LiveData]

I don't know if this is related, but I'm on Windows and when the Permission Set session expires, Python CDK often returns None for self.node.try_get_context('account'). I then have to rerun cdk, and that fixes the problem (which of course is annoying, especially since it takes so long to spin up JSII). That makes me think that maybe CDK is doing something in the wrong order.

[tim-finnigan]

I reached out to the SSO team to try and get more context on this issue. Still waiting to hear back but will update the issue when I have more information. I'll also link a couple more relevant documentation pages that I didn't see referenced yet:

CLI User Guide: https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html

If you are signed in to the sso-session you are updating, refresh your token by running the aws sso login command.

CLI command reference for aws-sso-login: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sso/login.html

Retrieves and caches an AWS SSO access token to exchange for AWS credentials. To login, the requested profile must have first been setup using aws configure sso. Each time the login command is called, a new SSO access token will be retrieved. Please note that only one login session can be active for a given SSO Session and creating multiple profiles does not allow for multiple users to be authenticated against the same SSO Session.

[benkehoe]

This is presumably a service-side issue. It's not possible for the CLI to update the token in the cache without updating the expiration. The cache entry is formed from the service response and then put in the cache in its entirety. So if there's a mismatch, it may be that the service (i.e., the sso-oidc API) does not have the same expiry window as the CLI and SDKs, so it is returning a token for the current (still-valid) session, with that session's expiration, rather than considering the session expired and initiating a new session.

[dtabuenc]

Yeah this happens all the time for us. Both in node sdk and also with cdk. Thanks to this issue we now have a work around of clearing the cookies and forcing a new session on the console to be generated. But this is obviously a mis-matched expiration issue and should not be this difficult to address.

[nwalters512]

I suspect that this might have something to do with the issue:

https://github.com/aws/aws-sdk-js-v3/blob/e503ab95f33cffc1f4a5452d79fbc410d3b6a32d/packages/credential-provider-sso/src/resolveSSOCredentials.ts#L15

If the credentials will expire within the next 15 minutes, they're flagged as already expired. If aws sso login doesn't use the same logic, it seems like there will be a 15-minute window during which aws sso login will refuse to update the non-expired credentials and the JS SDK will refuse to use the non-expired credentials.

[tnurmi82]

I just briefly had this same exact experience, when using git cli and attempting 'git pull'.
While issue was happening I stumbled here. After giving it some time, issue just went away, before I had the change to try cache clear or other workarounds.

aws --version

aws-cli/2.11.4 Python/3.11.2 Windows/10 exe/AMD64 prompt/off

[adcanis]

when running aws sso login --profile <your profile> a new set of credentials is generated making the ones in /.aws/credentials no good. Thus, when running say amplify push -y --profile <your profile> returns error "Failed to get profile credentials"

For now, I'm having to manually update the credentials after logging in via my login url before running aws sso login command for whatever account im in that day.

[thdxr]

I suspect that this might have something to do with the issue:

https://github.com/aws/aws-sdk-js-v3/blob/e503ab95f33cffc1f4a5452d79fbc410d3b6a32d/packages/credential-provider-sso/src/resolveSSOCredentials.ts#L15

If the credentials will expire within the next 15 minutes, they're flagged as already expired. If aws sso login doesn't use the same logic, it seems like there will be a 15-minute window during which aws sso login will refuse to update the non-expired credentials and the JS SDK will refuse to use the non-expired credentials.

incredible - this has been driving me crazy forever

found that if I logout of SSO in the browser it'll force a refresh - only way to get out of that limbo

[benkehoe]

There is so much noise and incorrect information in this thread. To clarify what's happening and why this isn't a bug in the CLI (but rather a mismatch between some SDKs and the Identity Center service—which still needs to be resolved!) let me run through the entire process:

1. When you call aws sso login for the first time (no active session with the service or cached credentials on your system), it calls customizations.utils.do_sso_login() with force_refresh=True. do_sso_login() calls botocore.utils.SSOTokenFetcher.fetch_token(), passing through force_refresh (set to True).
2. SSOTokenFetcher.fetch_token() just calls SSOTokenFetcher._token() With force_refresh set to True, SSOTokenFetcher._token does not check the cache at all. Instead, it goes directly into the SSOTokenFetcher._poll_for_token() method.
3. SSOTokenFetcher._poll_for_token() dispatches to the browser through the _on_pending_authorization() method which is set back in do_sso_login() unless you've given the --no-browser flag, in which case it's set earlier in the login command. The URL for popping up the browser comes from the verificationUriComplete property in sso-oidc.StartDeviceAuthorization API call
4. SSOTokenFetcher._poll_for_token() then starts polling for the token using the sso-oidc.CreateToken API call in the SSOTokenFetcher._create_token_attempt() method. CreateToken raises an error before you have logged in.
5. In the browser, you log in. This creates multiple cookies in your browser.
6. Once you've logged in, CreateToken returns successfully with an access token.
7. When that returns with an access token, it creates the "token" as a dict containing the access token and other fields, including the expiration date, purely from the API response (with one slight caveat, the response has a duration, expiresIn, and that's added to the system's current time to get a datetime expiresAt, but that is not the source of the problem here).
8. The token dict gets returned from _create_token_attempt() to _poll_for_token() and then to _token, which caches it and returns it (the return value isn't actually used in the login process).

Ok, so now you've got a valid token on your system. Let's look at what happens when attempt to make an API call with the CLI, like aws sts get-caller-identity. This is in botocore, used by both boto3 sessions and the AWS CLI when making API calls. We will see that the AWS CLI and boto3 do not suffer from the problem described in this issue.

9. I'm not going to go through the convoluted set up of botocore credential chains, but the relevant way that botocore sessions use SSO credentials is with botocore.credentials.SSOProvider. SSOProvider is set up with a botocore.tokens.SSOTokenProvider that it passes to a botocore.credentials.SSOCredentialFetcher that's used for profiles using SSO session config, and a botocore.utils.SSOTokenLoader that's used for profiles with inline SSO config.
10. The credentials returned by SSOProvider.load() are wrapped in DeferredRefreshableCredentials, which is a lazy-loading subclass of botocore.credentials.RefreshableCredentials, which does some complicated logic around thread-safe refreshing, but the main point is, when the credentials (the access key/secret key/session token kind, not the SSO token kind) are within 15 minutes of expiring (which again, is a different expiration than the one for the SSO token used to get these credentials), it will try to refresh these credentials using the token.
11. That refresh involves a call to SSOCredentialFetcher.fetch_credentials(), which is implemented in its parent class, botocore.credentials.CachedCredentialFetcher.fetch_credentials().
12. That involves a call to SSOTokenProvider.load_token(). Again, some botocore magic around lazy loading, but the token loading itself is in SSOTokenProvider._refresher().
13. Here, finally, is where the token expiration window is checked. The window is set to 15 minutes. Note that if it's inline SSO config, the SSOTokenLoader is used and there's no expiration check.
14. If the expiration is within (or after) the window, SSOTokenProvider calls sso-oidc.CreateToken with the cached refresh token.
15. If the CreateToken call is successful, it will create a new token dict, including with the returned refresh token and the expiration as provided in the response. If it's not successful, the existing token dict is used.
16. The token dict is used by SSOCredentialFetcher to call the sso.GetRoleCredentials API to get the familiar access key id, secret access key, and session token credentials. Caching for these is managed separately.
17. The error message "The SSO session associated with this profile has expired..." comes from UnauthorizedSSOTokenError, which is raised if GetRoleCredentials is not successful. Note that all errors from attempting to refresh the token are swallowed and the token refresher doesn't use an expiration window when raising an expiration error.

So while it will attempt to refresh the token before the expiration, but the only thing that actually says "this token is expired" when you call AWS with the token to get credentials for a specific account and role and AWS responds that the token is invalid, which only happens when the session (and token) is actually expired (which is the behavior you'd expect and want).

As @nwalters512 pointed out, the JS SDK is too eager on this front. Instead of using that 15 minute window to attempt credential refresh, it just considers it expired and does not attempt to get credentials. As per aws/aws-sdk-java-v2#3679, the Java SDK has the same logic.

To see why aws sso login can't fix this: If you call aws sso login again, even if your token is 5 minutes old and completely valid, that force_refresh=True means it will go through steps 1-8 again. There's no check in that process for expiration window.

When aws sso login is called before the session has expired, it dispatches to the browser. The browser has cookies for a session that's still valid; Identity Center doesn't ask you to authenticate, because you've got a valid session. This is a pretty reasonable thing to do! So then you've gone through that, and when CreateToken returns a (new) token back to the CLI, it has the same expiration as the old token, because it's for the current session with the current session's expiration.

Only once the browser does not hold cookies for a valid session (either because the cookies are gone or the session is no longer valid) will Identity Center re-authenticate you, creating a new session with a new expiration, and then the token returned by CreateToken will have that new expiration.

This is why logging out in your browser works; there's no longer a valid session to use, so when aws sso login dispatches to your browser, you log in and that creates a brand new session, and so CreateToken returns a token for that new session with the associated new expiration time.

So the mismatch here is between SDKs (the JavaScript v3 SDK, the Java SDK, and possibly others) that refuse to use a token that is within 15 minutes of its expiration, and the Identity Center sso-oidc service, which refuses to trigger a full re-authentication flow—which would then create a new session and issue a fresh token for the new session—before the session actually expires (and provides no mechanism for manually triggering re-authentication).

[tim-finnigan]

A huge thanks to @benkehoe for identifying and explaining the issue. I think much of the original confusion here comes down to inconsistent behavior between SDKs, despite expected behavior on the CLI side. Because this is a cross-SDK issue, I'm going to transfer it to our shared SDK repository for tracking going forward.

We have raised this issue internally and currently are in discussions with the service team. Further investigation is needed to determine the best path forward. We will update this issue when we have more information. Thanks all for your patience and contributions here thus far.

[benkehoe]

For the record, my position is that SDKs should not refuse to use a token that hasn't expired yet. An expiration window is great for refreshing credentials so that there's no gap where the SDK only has expired credentials, but it just doesn't make sense to be like "well, this might not work in 15 minutes, so better stop using it now".

[spoco2]

Just want to raise here that the issue mentioned ☝️ regarding cdk cli seemingly never triggering any sort of refresh, and so always having session timeouts set to 1 hour has been linked to this as the root cause.

Want to ensure that it's not a related issue that may not be fixed with any fix created for this. Other threads had surmised that using the aws cli triggered refreshes of the tokens as required, but the cdk cli didn't use aws cli and so didn't trigger any of the required token refreshes, and so was always only getting the initial 1 hour session, regardless of what the session length was set to against the Permission Set associated with the user and account they were logging into via SSO.

Thank you to @benkehoe for the detailed breakdown of the flows involved.

If anyone happens to know of any workarounds until this is solved, it would be wonderful.

The issue of there being a 15 minute window of the system saying the token is invalid, but also not allowing a new one to be refreshed is not what we're hitting, but rather our sessions being always 1 hour long no matter what I have them set to within the Permission Set (12 hours).

[gastonsilva]

@spoco2 a workaround I found is to run any command with aws-cli on the sso-session profile before using cdk (something simple like aws s3 ls --profile my-sso-session-profile), this performs a renovation and avoids the need to make a full sso-login which might be slower. There are several ways to do this depending on your project structure and language.
For example you can include a line in your cdk/bin to spawn the command, if using npm adding the AWS command to a script, etc.
Hope this helps!

[spoco2]

@gastonsilva , thanks. I was thinking something like that, or a cron job to do that every 10 minutes or so would be something to keep it alive in the meantime!

[tim-finnigan]

Noting here that this has been fixed in Java SDK v2.20.100: aws/aws-sdk-java-v2#3679 (comment)

Since this is no longer a cross-SDK issue but a JavaScript SDK issue, I think we should continue tracking here going forward:

• AWS SDK is unable to fetch AWS credentials from AWS IAM Identity Center if the Identity Center access token for an SSO session expiry time is between 5 and 15 minutes from now aws-sdk-js-v3#4798
• SSO Credentials Provider fails to load AWS credentials if the AWS IAM Identity Center access token associated with an SSO session has expired or is about to expire aws-sdk-js#4441

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[benkehoe]

@tim-finnigan Just to confirm, have all the other AWS SDKs been checked for this problem?

[tim-finnigan]

@benkehoe Yes we had members from each SDK team review and confirm, only the Java and JavaScript SDKs were found to have this issue.

[jamsmd]

I had the same issue. I'm using ARC browser and the aws sso login command triggered mini arc and whenever I finish the auth process in mini arc I would get the error.

Whenever the aws sso login command opens mini arc click the right top to open the auth proces in regular arc.

That did the trick for me!

[Nadinee77]

Does anyone know anything about when the sso session will expire