Env TF_VARS are not working and docs are outdated

[reinis-s]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

cdktf & Language Versions

cdktf debug
language: python
cdktf-cli: 0.15.1
node: v18.13.0
cdktf: 0.15.1
constructs: 10.1.231
jsii: 1.73.0
terraform: 1.3.7
arch: arm64
os: darwin 21.4.0
python: Python 3.10.9
pip: pip 22.3.1

Issue

1. TF_VAR and documentation
2. How to pass dynamic input without restoring to create separate var file or using var as arguments?

I'm not sure was there as good reason to disable TF_VAR_* but documentation still says that we can use TF_VAR_.

What's the team current suggestion how dynamically inject config into CDKTF?

References

• #1714
• #2342

[ansgarm]

Hi @reinis-s 👋

How are you using values set via TF_VAR_x? #1714 only removed them from the synth step to not accidentally expose secrets – they should be available in the plan/apply phase (when Terraform is invoked), if not, that's a bug.

[reinis-s]

Hey thank you for swift response: diff deploy fails with missing vars that are set by TF_VAR

#.envrc set env var
export TF_VAR_name="my_stack"

❯ pipenv run env  | grep -i TF_VAR
TF_VAR_name=my_stack

# main.py

#!/usr/bin/env python
from constructs import Construct
from cdktf import App, TerraformStack, TerraformOutput, TerraformVariable
from imports.aws.provider import AwsProvider
from imports.my_stack import MYStack


class MyStack(TerraformStack):
    def __init__(self, scope: Construct, id: str):
        super().__init__(scope, id)

        AwsProvider(self, "aws")
        name = TerraformVariable(self, "name", type="string")
        localModule = MyStack(self, name=name)

app = App()
MyStack(app, "my_stack")
app.synth()

❯ cdktf deploy
...
my_stack  Success! Terraform has validated the lock file and found no need for changes.
my_stack  Missing variable: 'main_name_27FB85A4'. You can provide it using the 'TF_VAR_main_name_27FB85A4' environment variable.

0 Stacks deploying     1 Stack done     0 Stacks waiting
Invoking Terraform CLI failed with exit code 1

❯ cdktf diff
my_stack  Initializing modules...
my_stack  Initializing the backend...
my_stack  Initializing provider plugins...
...
my_stack  Success! Terraform has validated the lock file and found no need for changes.

⠏  Processing
[2023-01-27T19:34:26.247] [ERROR] default - ╷
│ Error: No value for required variable
│
│   on cdk.tf.json line 98, in variable:
│   98:     "main_name_27FB85A4": {
│
│ The root module input variable "main_name_27FB85A4" is not set, and has
│ no default value. Use a -var or -var-file command line argument to provide
│ a value for this variable.
my_stack  Missing variable: 'main_name_27FB85A4'. You can provide it using the 'TF_VAR_main_name_27FB85A4' environment variable.

[reinis-s]

One more thing, I've no idea if expected or separate bug
to define var I have to use

name = TerraformVariable(self, "name", type="string").string_value

If don't supply correct value type for every var - .string_value,.list_value etc.. cdktf fails with

TypeError: type of argument name must be str; got cdktf.TerraformVariable instead

[reinis-s]

Hey @ansgarm please let me know if you need any additional info.

[ansgarm]

Hi @reinis-s 👋

The problem seems to be that your variable gets assigned the name main_name_27FB85A4, so you'd need to use TF_VAR_main_name_27FB85A4 to set it.

However, if the variable is defined in the root of the stack, it shouldn't get that name nowadays. What does your cdktf.json configuration file look like? Does it include a excludeStackIdFromLogicalIds property? If yes, what does that one say?

[reinis-s]

@ansgarm
I've imported custom module that written hcl by me.
This module do have variable(but it doesn't have random id or prefix main_ :

variable "name" {
  type = string
}

cdktf.json

{
    "language": "python",
    "app": "pipenv run python main.py",
    "projectId": "e6cdf273-a12e-4391-a940-599259475371",
    "sendCrashReports": "false",
    "terraformProviders": ["aws@~> 4.0"],
    "terraformModules": [
      {
        "name": "vpc",
        "source": "terraform-aws-modules/vpc/aws",
        "version": "~> 3.0"
      },
      {
        "name": "my_stack",
        "source": "git::https://github.com/MY_REPO//modules/my_stack?ref=master"
      }
    ],
    "codeMakerOutput": "imports",
    "context": {
      "excludeStackIdFromLogicalIds": "true",
      "allowSepCharsInLogicalIds": "true"
    }
  }
  

[ansgarm]

Hm, that's looking good so far.
Could you paste your main.py? Or link a snippet, if it is long?

[reinis-s]

I've already pasted couple comments above? #2550 (comment)
Happy to provide more details if this not enough

[ansgarm]

Hm, I'm only seeing a main.tf there – is that from your main.py?
I'd require some more details from that file – e.g. what is self referencing to in your Terraform Variable?
Did you nest code in custom constructs?

Maybe pasting the complete file would help – feel free to delete resources that seem superfluous but the general structure is required for me to assess the problem.

[reinis-s]

I've updated main.py code to have full version and included module
Terraform module mystack:

# main.tf
terraform {
  required_version = ">=1.1.7"
}

resource "aws_route53_zone" "zone" {
  name          = var.name
  force_destroy = true
}

# variables.tf
variable "name" {
  type = string
}

# provaider.tf
provider "aws" {
  alias   = "shared"
  region  = var.aws_shared_account_region
  profile = var.aws_shared_account_profile
}

[ansgarm]

Thanks! This helps a bit, but I'm a bit confused about this part:

class MyStack(TerraformStack):
    def __init__(self, scope: Construct, id: str):
        super().__init__(scope, id)

        AwsProvider(self, "aws")
        name = TerraformVariable(self, "name", type="string")
        localModule = MyStack(self, name=name) # <-- wouldn't this recursively include MyStack within itself?

How does your generated cdktf.out/stacks/my_stack/cdk.tf.json look like?

[reinis-s]

Ahh sorry my bad, that's typo - I remove company name and accidentally replaced with the same name.

from imports.foo_stack import FooStack

class MyStack(TerraformStack):
    def __init__(self, scope: Construct, id: str):
        super().__init__(scope, id)

        AwsProvider(self, "aws")
        name = TerraformVariable(self, "name", type="string")
        localModule = FooStack(self, name=name)

[ansgarm]

Is there a TerraformVariable construct defined in FooStack?

[reinis-s]

No as far as I know - FooStack is just imported terraform module written in hcl aka imported
This comment contains roughly what's inside:
#2550 (comment)

Basic setup:

1. Import terraform module written in hcl
2. try to supply module inputs vars via TF_VAR_*
3. My guess that for some reason imported hcl module generate custom vars names such as main_name_27FB85A4 instead of re-using what's in hcl name.

[ansgarm]

Hm, I didn't know it was possible to supply input vars to modules from the outside. You'd need to pass them to your FooStack construct instead.

[ansgarm]

At this point, it would really help, if you could publish a minimal example to e.g. a Github repository that I can use to reproduce the issue.

[kemalizing]

Hi,
I have the same issue with java (kotlin) TerraformVariable. its name is converted to <stack-name-lowercase without any special char>_<TerraformVariable-name-lowercase without any special char>_<something like a hashcode>.
As a result I can't assign var value using env settings.

I didn't import my stack from HCL but just wrote natively using the java lib.

Any improvement on this issue?

[kemalizing]

here is my code:

fun main() {
    val app = App()

    TerraformVariable.Builder.create(app, "app_client_secret")
        .type("string")
        .sensitive(true)
        .build()

    SomeStack(app)

    app.synth()
}

class SomeStack(scope: Construct) : TerraformStack(scope, "ctmx") {
    init {
        TerraformVariable.Builder.create(this, "stack_client_secret")
            .type("string")
            .sensitive(true)
            .build()
    }
}

and here is the generated json:

{
  "//": {
    "metadata": {
      "backend": "local",
      "stackName": "ctmx",
      "version": "0.16.0-pre.43"
    },
    "outputs": {
    }
  },
  "terraform": {
    "backend": {
      "local": {
        "path": "/Users/ke/IdeaProjects/ctmx/ctd-infra/terraform.ctmx.tfstate"
      }
    }
  },
  "variable": {
    "ctmx_stackclientsecret_274F4C75": {
      "sensitive": true,
      "type": "string"
    }
  }
}

As you see if I supply the app as the scope parameter for TerraformVariable then it doesn't create any elements in json. On the other hand, if I supply the stack as the scope parameter, then it renames the parameter which breaks everything.

Lastly, in my cdktf.json, I have these settings which don't have any impact:

  "context": {
    "excludeStackIdFromLogicalIds": "true",
    "allowSepCharsInLogicalIds": "true"
  }

[kemalizing]

1 extra info about this. I tried the example from your github repo:

terraform-cdk/examples/java/documentation/src/main/java/com/mycompany/app/MainHCL.java

Line 26 in e8ad548

TerraformVariable petNameLength = new TerraformVariable(this, "petNameLength", TerraformVariableConfig.builder()

the resulting variable in json is this:

  "variable": {
    "mainhcl_petNameLength_9F384CC9": {
      "default": 2,
      "description": "Pet name length",
      "type": "number"
    }
  }

Even the example in TF repo changes the var id while in the documentation it mentions that we can set the value of the variable using environment variable TF_VAR_<the id we set for the variable>. Unfortunately, that is not working...

[kemalizing]

One more thing. It is not just about variables, synth changes all the names of all the terraform resources...

[kemalizing]

Another finding: this doesn't happen in ts language. happens in java, also happened in python for @reinis-s

[jsteinich]

@kemalizing when observing these findings, are you running the application directly or running via cdktf synth? Running directly ignores that context (see #798).

If the scope isn't the root stack, then the hash will still be appended. It is possible to use the overrideLogicalId function to force a name as well.

[kemalizing]

hi @jsteinich
thanks for your reply. Indeed, I'm running the app directly. I didn't want to install a cli on my pipeline to generate tf.

Do you know if there is any improvement on the issue that you linked? As mentioned there, having different names doesn't make any sense.

Looks like, currently the only possible solution is the one you suggested, using overrideLogicalId.

[github-actions]

Hi there! 👋 We haven't heard from you in 30 days and would like to know if the problem has been resolved or if you still need help. If we don't hear from you before then, I'll auto-close this issue in 30 days.

[github-actions]

I'm closing this issue because we haven't heard back in 60 days. ⌛️ If you still need help, feel free to comment or reopen the issue!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.