Saving state file can fail when STS tokens expire during run

[bigkraig]

We're using STS tokens that we need to rotate once an hour. Sometimes TF configs can take a long time (i'm looking at you, RDS) and very bad things happen if the token expires before TF can save the state file to S3.

Simply reloading the profile and trying the write again would resolve this, but I am not sure if this is the most elegant way. An error showing where your state file is and letting you know its not saved to the remote location would be an improvement.

[radeksimko]

I'm afraid there isn't any easy way to prevent this if the token is generated outside Terraform, since Terraform can't know when is the token going to expire - correct me if I'm wrong. I couldn't find any useful STS API method for this.

When Terraform gets support for real AssumeRole* (it is surely tracked somewhere under a different issue), then it may be either gathering new credentials for each run or cache those somewhere and then we could introduce something like this:

provider "aws" {
  sts_token_min_validity = "30min"
}

[bigkraig]

Refreshing the AWS service should re-read the credentials file that pulls the token in. We're using ADFS for our federated logins which means a NTLM handshake that isn't supported by golang yet, and I imagine there are other methods out there that are equally difficult to support. It's probably much easier to assume the user has an updated token in their ~/.aws/credentials waiting to be used. We run a tool that generates new tokens for us every 55 minutes.

On Apr 1, 2016, at 5:55 AM, Radek Simko notifications@github.com wrote:

I'm afraid there isn't any easy way to prevent this if the token is generated outside Terraform, since Terraform can't know when is the token going to expire - correct me if I'm wrong. I couldn't find any useful STS API method for this.

When Terraform gets support for real AssumeRole* (it is surely tracked somewhere under a different issue), then it may be either gathering new credentials for each run or cache those somewhere and then we could introduce something like this:

provider "aws" {
sts_token_min_validity = "30min"
}
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub #5927 (comment)

[apparentlymart]

Hi @bigkraig! Sorry for the long silence here.

I just wanted to clarify what you were suggesting in your most recent comment. It sounds like in your environment you have a separate tool that periodically updates what's in ~/.aws/credentials as sessions expire, and so you want Terraform to try to re-read this file if it encounters an authentication error. Did I understand that correctly?

It might be hard for us to implement something like that since we generally delegate all of the auth details to the AWS SDK, but if this is a common pattern in other AWS-related tools then perhaps there's a way to ask the AWS SDK to support this behavior.

[bigkraig]

@apparentlymart at first I thought April 1st wasn't that long ago!

That is a correct understanding.

I'd assume any enterprise using SAML authentication to AWS has some similar tool written, which should be a lot. Most security teams realize the problems with a lot of users having a lot of keys that can be used from or leaked to the internet. Getting it fixed in the SDK is a great idea. I can send suggestions from my side to our AWS reps and see if we can get something happening in the client.

[apparentlymart]

@bigkraig if we were to make sure Terraform can correctly support sts:AssumeRole itself, would that be a suitable/better way to solve the problem? I think that would be easier to achieve since the necessary support is definitely already in the SDK, including auto-refreshing, and there are already issues open for this and some progress already made thanks to great work by @radeksimko .

[apparentlymart]

Talking of which, by coincidence today @jbardin was working on #13384 which should make the S3 remote backend have the same authentication behavior as the AWS provider itself. I'm not sure if the AssumeRole capabililties on the AWS provider are sufficient for your use-case, but if so the S3 backend should now be too.

[danvaida]

any updates on this one? same issue here with ADFS (even when the remote state backend is not S3).

[apparentlymart]

Hi all! Sorry for the long silence here.

The S3 backend has had support for a role_arnsetting for some time now, which addresses this problem by having Terraform itself call sts:AssumeRole and call it again if the role expires during the run. This is using the renewal mechanism built in to the AWS SDK behind the scenes.

Since that provides a solution to the expiring tokens, I'm going to close this out.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.