Backends - ability to use credentials fetched from Vault

[mtarnoci]

Terraform Version

[terraform_svc@mag-a-vcter002 11fbb880]$ terraform version
Terraform v1.3.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.19.0
+ provider registry.terraform.io/hashicorp/null v3.1.1
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/time v0.7.2
+ provider registry.terraform.io/hashicorp/vault v3.8.2
[terraform_svc@mag-a-vcter002 11fbb880]$

Use Cases

When configuring Terraform backends, for example S3, it is not possible to use data fetched from the Vault in Credentials configuration.
Use case: to avoid to use credentials stored in .tf file in plain text.

Attempted Solutions

terraform {
  backend "s3" {
    region                = "eu-central-1"
    key                   = "terraform.tfstate"
    encrypt               = true
    bucket                = "test-backend"
    workspace_key_prefix  = "test-key-prefix"

    
    
    access_key = data.vault_generic_secret.cor_prov_user.data["AWS_ACCESS_KEY_ID"]
    secret_key = data.vault_generic_secret.cor_prov_user.data["AWS_SECRET_ACCESS_KEY"]

    role_arn     = data.vault_generic_secret.customer_role.data["role_arn"]
    external_id  = data.vault_generic_secret.customer_role.data["external_id"]
    session_name = "session_name"

  }
}

Error: Variables not allowed
│ 
│   on 00_backend_s3.tf line 15, in terraform:
│   15:     access_key = data.vault_generic_secret.cor_prov_user.data["AWS_ACCESS_KEY_ID"]
│ 
│ Variables may not be used here.
╵

╷
│ Error: Variables not allowed
│ 
│   on 00_backend_s3.tf line 16, in terraform:
│   16:     secret_key = data.vault_generic_secret.cor_prov_user.data["AWS_SECRET_ACCESS_KEY"]
│ 
│ Variables may not be used here.
╵

╷
│ Error: Variables not allowed
│ 
│   on 00_backend_s3.tf line 22, in terraform:
│   22:     role_arn     = data.vault_generic_secret.customer_role.data["role_arn"]
│ 
│ Variables may not be used here.
╵

╷
│ Error: Variables not allowed
│ 
│   on 00_backend_s3.tf line 23, in terraform:
│   23:     external_id  = data.vault_generic_secret.customer_role.data["external_id"]
│ 
│ Variables may not be used here.

Proposal

No response

References

No response

[apparentlymart]

Thanks for sharing this use-case, @mtarnoci!

A typical way to solve the problem of integrating external credentials sources with AWS is to specify an external credentials process in your AWS configuration.

The S3 backend and the hashicorp/aws provider both use the AWS SDK for authentication and so should support this setting. I don't know if there is an existing credentials program for integrating with Vault or if you would need to write one yourself, but you can see in the documentation I linked to how the AWS SDK expects that external program to behave.

We typically expect backends and providers to be authenticated via the target platform's usual mechanisms, rather than within the Terraform language itself, because that then minimizes the exposure of your credentials to your Terraform modules where they might, for example, be accidentally shown onscreen as part of a plan.

The S3 backend and hashicorp/aws provider both aim to support the standard AWS credentials configuration options so that you can set up your AWS configuration just once and then use it across both Terraform and the AWS CLI, and any other software which integrates with AWS using the SDK.

A typical Terraform configuration should not set access_key or secret_key settings in the configuration of either backend "s3" or provider "aws". Those settings are there as a last resort for more complicated situations that the AWS configuration file format cannot represent.

[wxGold]

Same use-case here way with Vault and AWS creds, come across this prob with s3 backend.
with data.vault... no need to have it configured in any config or credentials file, which is better.
We must have this option also for the s3 backend auth please

meanwhile, @mtarnoci did you manage some workaround for that? except of providing as env vars or anything related to .config or .credentials files (profile names of aws configure)

[uo-thomas]

I have the same use case. You can interpolate access credentials in the AWS provider, so why not with the S3 backend, especially if the use case is to use another Hashicorp product (in this case Vault) configured with an AWS Secrets backend for dynamic IAM provisioning. No need to store credential configurations either in the Terraform project or in plain text in configuration files on the machine.

[liquidspikes]

Same need, would love for this to work. :)

[gdavison]

It looks like you're trying to use values retrieved from a Terraform data source (vault_generic_secret) to configure the S3 backend. @apparentlymart could clarify, but I don't believe that that is supported by Terraform.

[uo-thomas]

It looks like you're trying to use values retrieved from a Terraform data source (vault_generic_secret) to configure the S3 backend. @apparentlymart could clarify, but I don't believe that that is supported by Terraform.

I think we all know that isn't supported, hence the issue / feature request. We would simply like the S3 backend block to behave the same as a providers block.

[gdavison]

OK, I'll update the issue. This isn't related to the S3 Backend specifically, but to any backend and how they can be configured.

[manish-jangra]

much needed feature. Any update on this @crw

[ehuizar1028]

Any update on this? This would be a very useful feature for us :)

[crw]

No update, the information and recommendations in #31869 (comment) are still applicable.

Note, may also be related to #13022.