Support option to output stderr regardless of failure

[skeggse]

Terraform Version

Terraform v0.14.5
+ provider registry.terraform.io/hashicorp/aws v3.34.0
+ provider registry.terraform.io/hashicorp/external v2.1.0

Affected Resource(s)

• external

Terraform Configuration Files

data "external" "test" {
  program = [
    "bash", "-ec",
    <<-EOF
    echo script making progress >&2
    echo '{}'
    # exit 1  # stderr gets forwarded to console when exit code != 0
    EOF
  ]
}

Debug Output

Plan

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Panic Output

N/a

Expected Behavior

terraform {plan,apply} should (optionally) print the stderr output even when the exit code from the script is zero.

Actual Behavior

terraform {plan,apply} does not print the stderr output unless the exit code is nonzero.

Steps to Reproduce

1. terraform apply or terraform plan

Important Factoids

N/a

References

scottwinkler/terraform-provider-shell#27, maybe

[nfroidure]

I think it would help me to solve that issue : https://stackoverflow.com/questions/66490152/how-to-debug-terraform-external-providers-with-concurrency-issues

Indeed, I can find any bad JSON output in the external programs I run and I'm stuck with no information on the output Terraform takes in count or any error that would make my command to fail from sometimes.

[bflad]

Hi @skeggse 👋 Thank you for raising this.

Terraform Providers have a few ways to raise execution output:

• Log messages at trace, debug, info, warn, or error levels. These are not visible by default in the user interface and typically inspected by creating a log file that captures these (although that is technically not a requirement). See also: https://www.terraform.io/internals/debugging
• Warning diagnostics, which are visible in the user interface, however they do not prevent further Terraform execution.
• Error diagnostics, which are visible in the user interface and prevent further Terraform execution.

In this scenario, it appears that either of the first two options (log messages or warning diagnostics) are the most viable since raising an error diagnostic would be a drastic behavior change. Personally, I do not think warning diagnostics are a good fit here though because practitioners will not have a good way to exclude this information and see this as a user experience breaking change for programs that have previously worked without this additional output for a long time.

This data source has had some recent logging improvements that will be released in the next version (#95). This logging could potentially be further enhanced by including stderr if present. Would adding stderr output (if present) to that trace log suit this feature request?

See also hashicorp/terraform#17267 in the Terraform CLI issues for another potential idea here, which cannot be implemented today by providers.

[skeggse]

Warning diagnostics, which are visible in the user interface, however they do not prevent further Terraform execution.

My primary use-case is for external data sources in either root workspaces themselves, or in internal/organization-specific modules. It's been a while since I filed this issue, so I don't remember all of the context. However, my guess about my original motivation is that it's easy for engineers who are not particularly familiar with Terraform to give up after a minute or so, but operations performed by external data sources may take longer and/or may require user input in an external system, and prompting them to attend to that action is currently not feasible within Terraform. hashicorp/terraform#17267 would certainly go a long way toward improving that. However, for some use-cases, it would be nice if additional access to warning output *perhaps opt-in?) could be afforded for particular use-cases where the author and user are within the same organization.

[austinvalle]

v2.3.3 of the external provider has been released and will now output stderr logs regardless of the exit code returned by the program. These will all show up on the TRACE log level: https://developer.hashicorp.com/terraform/internals/debugging

If you have more specific asks for handling of this situation outside of logs, feel free to submit a new issue. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.