Cannot refresh/plan using vault generated STS Federation Token creds for AWS

[Yuxael]

Terraform Version

Terraform v0.10.0

Terraform Configuration Files

data "vault_generic_secret" "aws_creds" {
  path = "aws/sts/deploy"
}


provider "aws" {
    region = "${var.ec2_region}"
    access_key = "${data.vault_generic_secret.aws_creds.data["access_key"]}"
    secret_key = "${data.vault_generic_secret.aws_creds.data["secret_key"]}"
    token = "${data.vault_generic_secret.aws_creds.data["security_token"]}"
}

Debug Output

https://gist.github.com/Yuxael/9f3c0ff87f26053688a3bb6d8420877d

Expected Behavior

AWS credentials regenerated and terraform refresh/plan should run successfully

Actual Behavior

Just hangs for eternity. It keeps looping over dag/walk: vertex ... shown at the end of debug.

Steps to Reproduce

1. Run 'terraform refresh/apply(whatever that generates/updatestfstate`, should work first time).
2. Wait until lease time for STS token expire.
3. Run again terraform refresh/plan (will hang forever - cancel).
4. Manually remove data.vault_generic_secret.aws_creds section from tfstate.
5. Run once again terraform refresh/plan (will work this time).

Important Factoids

When running first time it runs successfully and do refresh tfstate or outputs a plan. Reruns also works fine until lease time for credentials expires, then it behaves like debug shows.

What bothers me is also the fact that is saves credentials generated by Vault in tfstate:

"data.vault_generic_secret.aws_creds": {
                    "type": "vault_generic_secret",
                    "depends_on": [],
                    "primary": {
                        "id": "0d3e2d54-5e3f-a933-3d2c-a2f8e4a29d34",
                        "attributes": {
                            "data.%": "3",
                            "data.access_key": "<redacted>",
                            "data.secret_key": "<redacted>",
                            "data.security_token": "<redacted>",
                            "data_json": "{\"access_key\":\"<redacted>\",\"secret_key\":\"<redacted>\",\"security_token\":\"<redacted>\"}",
                            "id": "0d3e2d54-5e3f-a933-3d2c-a2f8e4a29d34",
                            "lease_duration": "3599",
                            "lease_id": "aws/sts/deploy/c9808580-0640-b8f2-eaeb-0c6db0997dd5",
                            "lease_renewable": "false",
                            "lease_start_time": "RFC7779",
                            "path": "aws/sts/deploy"
                        },
                        "meta": {},
                        "tainted": false
}

This is talked about in issue #516 and is directly connected with this problem because in order to make plan and refresh commands work again I have to manually delete those stale credentials from tfstate.

References

• Storing sensitive values in state files #516

[jbardin]

Hi @Yuxael,

Thanks for filing such a thorough issue.

I'm going to tag this for the vault provider for now. It seems that the vault data source is being read correctly, but the credentials aren't being refreshed.

[hashibot]

This issue has been automatically migrated to hashicorp/terraform-provider-vault#9 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to this issue and let us know.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.