Are OTF (env) vars forwarded to OTF agents?

[pat-s]

When executing a workspace run through either "Remote" or "Agent" I get an error for the latter stating that a specific terraform variable could not be found/is missing.

Rerunning the very same state with "Remote" works just fine.

I am wondering: is there maybe an issue which prevents the terraform var (defined in OTF) from being honored when using the "agent" option? On the first look it seems like it's not being exported to the runner.

I also don't have an idea how to debug this properly. Any pointer is welcome :)

[pat-s]

Could it be that there is some dedicated action required to forward the variables in OTF to the agents? From my current experience it seems that no variables are existent.
Yet I don't see any pointer in the docs to set something up.

Probably missing something obvious since otherwise many people would have reported this way earlier 🤔

[leg100]

I've just tried with a single workspace variable and the variable is coming through okay. (Having said that, I recently refactored variable handling in order to introduce variable sets, so I wouldn't be surprised if there are some regressions).

You'll have to give me information for me to debug:

• your terraform configuration
• all variables defined, both sets and workspace variables
• logs from both otfd and otf-agent
• turn verbosity of logs to 9 for both: -v 9
• enable http request logging: --log-http-requests

[pat-s]

Will do!

I've just updated both the server and agent to 0.1.10 and faced another issue

Error: downloading zipfile: received non-200 HTTP code: 404

This happens if either the server or agent is refreshed and the respective other stays untouched. Refreshing the other one as well helps to resolve it.

FYI: --log-http-requests is not documented in the docs, is it a hidden flag? For both server and agent or only for one? I've set if for both but don't see any HTTP logs incoming.

You'll have to give me information for me to debug:

I've done the config changes and hope the following helps somehow:

• No var sets, only single workspaces vars
• Tried with both normal env var and terraform var

server

[lots of repetitions from the logs below]

2023/09/09 07:50:52 DEBUG-2 written logs id=run-ctA7fUZPDIb0obq4 phase=plan offset=11895
2023/09/09 07:50:52 INFO request duration=8ms status=200 method=PUT path="/api/v2/runs/run-ctA7fUZPDIb0obq4/logs/plan?offset=11895"
2023/09/09 07:50:52 DEBUG-2 written logs id=run-ctA7fUZPDIb0obq4 phase=plan offset=12002
2023/09/09 07:50:52 INFO request duration=7ms status=200 method=PUT path="/api/v2/runs/run-ctA7fUZPDIb0obq4/logs/plan?offset=12002"
2023/09/09 07:50:52 DEBUG-8 retrieved agent token organization=<org> id=at-T6kjov7j4nnJ9cF9
2023/09/09 07:50:52 DEBUG-2 written logs id=run-ctA7fUZPDIb0obq4 phase=plan offset=12536
2023/09/09 07:50:52 INFO request duration=9ms status=200 method=PUT path="/api/v2/runs/run-ctA7fUZPDIb0obq4/logs/plan?offset=12536"
2023/09/09 07:50:52 DEBUG-8 retrieved agent token organization=<org> id=at-T6kjov7j4nnJ9cF9
2023/09/09 07:50:52 DEBUG-2 written logs id=run-ctA7fUZPDIb0obq4 phase=plan offset=12826
2023/09/09 07:50:52 INFO request duration=19ms status=200 method=PUT path="/api/v2/runs/run-ctA7fUZPDIb0obq4/logs/plan?offset=12826"
2023/09/09 07:50:52 DEBUG-8 retrieved agent token organization=<org> id=at-T6kjov7j4nnJ9cF9
2023/09/09 07:50:52 ERROR creating report error="unexpected end of JSON input" id=run-ctA7fUZPDIb0obq4 phase=plan subject=at-T6kjov7j4nnJ9cF9
2023/09/09 07:50:52 INFO request duration=33498ms status=200 method=GET path="/app/runs/run-ctA7fUZPDIb0obq4/tail?phase=plan&offset=0"
2023/09/09 07:50:52 INFO finished plan id=run-ctA7fUZPDIb0obq4 resource_changes=+0/~0/−0 output_changes=+0/~0/−0 subject=at-T6kjov7j4nnJ9cF9 run_status=errored
2023/09/09 07:50:52 INFO request duration=89ms status=200 method=POST path=/api/v2/runs/run-ctA7fUZPDIb0obq4/actions/finish/plan?
2023/09/09 07:51:14 INFO request duration=0ms status=302 method=GET path=?
2023/09/09 07:51:14 INFO request duration=0ms status=302 method=GET path=/app/organizations?
2023/09/09 07:51:14 INFO request duration=0ms status=200 method=GET path=/login?
2023/09/09 07:51:19 INFO request duration=60005ms status=200 method=GET path="/app/runs/run-ctA7fUZPDIb0obq4/tail?phase=apply&offset=0"

agent

│ 2023/09/09 08:06:20 INFO stream update info="successfully connected"                                                                                                                                           │
│ 2023/09/09 08:07:11 INFO executing phase run=run-cjoYPmymkupqsGic phase=plan                                                                                                                                   │
│ 2023/09/09 08:07:43 ERROR executing phase run=run-cjoYPmymkupqsGic phase=plan error="1 error occurred:\n\t* exit status 1: Error: Error making API request. URL: PUT https:/<address>/v1/<path> │
│ /login/terraform Code: 500. Errors: * missing password with provider[\"registry.terraform.io/hashicorp/vault\"], on provider.tf line 22, in provider \"vault\": 22: provider \"vault\" {\n\n"                  │
│ 2023/09/09 08:07:43 INFO finishing phase run=run-cjoYPmymkupqsGic phase=plan 

In OTF I've tried both, env vars and TF vars.

In TF the var is then used as follows

variable "<VAR>" {
  sensitive = true
}

and then further as var.<VAR>. Again it works when simply switching to the "Remote" option, i.e. running it directly on the server host, without any other changes.

The workspace is using the "cloud" backend.

The TF code part which consumes the var looks as

provider "vault" {

  address = "<address>"
  auth_login {
    path = "<path>"

    parameters = {
      password = var.<VAR>
    }
  }
}

The agents run on k8s using https://github.com/pat-s/otf-agent-helm/. It might also be that there are additional config issues in the agent setup but the "only" important piece should be the actual connection between server and agent - which exists - via the OTF_TOKEN.

The final error then looks as

Planning failed. Terraform encountered an error while generating this plan.
 
╷
│ Error: Error making API request.
│
│ URL: PUT https://<address>/v1/<path>
│ Code: 500. Errors:
│
│ * missing password
│
│   with provider["registry.terraform.io/hashicorp/vault"],
│   on provider.tf line 22, in provider "vault":
│   22: provider "vault" {
│
╵

[leg100]

Ah, sensitive variable values are "scrubbed" before being sent over the wire. I'll need to change this accordingly.

[pat-s]

Thanks for checking!

I've just upgrade both server and agent to 0.1.12 and unfortunately I still face the same issues, both for normal env vars and TF env vars.

If there's any way I can help, let me know!

[leg100]

I suggest trying with the simplest configuration that demonstrates the problem, e.g.:

terraform {
  cloud {
    hostname     = "localhost:8080"
    organization = "acme"

    workspaces {
      name = "prod"
    }
  }
}

variable "foo" {
  default = "override_me"
}

output "foo" {
  value = var.foo
}

And then set a workspace variable for foo, with a different value. If that works then it suggests that the basic propagation of variables to agents is functioning okay. If so, then try altering your configuration until you can re-create the issue.

[pat-s]

Thanks. I tried that but this only helps for non-sensitive vars as the value will never be shown in the output module for sensitive ones.

Normal vars work just fine. But even after recreating the sensitive vars, these don't seem to arrive within the build:

Planning failed. Terraform encountered an error while generating this plan.
╷
│ Error: Error making API request.
│
│ URL: PUT https://vault.cynkra.com/v1/auth/userpass/login/terraform
│ Code: 500. Errors:
│
│ * missing password
│
│   with provider["registry.terraform.io/hashicorp/vault"],
│   on provider.tf line 22, in provider "vault":
│   22: provider "vault" {

When trying with non-sensitive vars and some dummy entries, I get a different error stating "invalid password" as expected. So the claim of "missing password" might really be an empty variable definition.

How did you check that sensitive vars make it through in #591? Is there a chance they still don't make it through when being sent to remote agents?

v0.1.12

[pat-s]

Works now! I had an issue updating my fork and merging in the respective commits which resolved the issue.

Totally a fault of mine! 🙏 thanks for the fix and sorry for the noise...