Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option for less verbose output #432

Closed
andrewhertog opened this issue Mar 7, 2018 · 16 comments · Fixed by #1510
Closed

Option for less verbose output #432

andrewhertog opened this issue Mar 7, 2018 · 16 comments · Fixed by #1510
Labels
enhancement New feature or request

Comments

@andrewhertog
Copy link

andrewhertog commented Mar 7, 2018

Is there an option that will hide a lot of the extra information that terragrunt outputs? While this information is nice to see, it would be nice to have a clean output that would just show us the raw output from terraform.

@brikis98 brikis98 added enhancement New feature or request help wanted labels Mar 8, 2018
@brikis98
Copy link
Member

brikis98 commented Mar 8, 2018

Not currently, but there should be. I'd welcome a PR to move Terragrunt to a proper logging system with INFO/DEBUG/etc and a way to set the log level via CLI. We've been meaning to move it over to gruntwork-cli as the base, which has that sort of logging built-in, but haven't had time. A PR for that is very welcome.

In the meantime, a workaround is to note that Terragrunt logs everything to stderr, so you could do terragrunt apply 2>/dev/null to suppress it.

@ezh
Copy link

ezh commented Oct 23, 2018

solution for bash only:

terragrunt apply 2> >(grep -v "\[terragrunt]" >&2)

STDOUT and STDERR lines are preserved except [terragrunt] ones

@josegonzalez
Copy link

josegonzalez commented Dec 20, 2018

One side-effect of this is that terragrunt output var - where var is a variable defined in your output - has a bunch of intermediate output, so you necessarily need to redirect stderr to ensure you're only getting what you want.

As well, if you're overriding the terraform-source, then there is extra output that is added to stdout, making terragrunt output var useless.

@brikis98
Copy link
Member

brikis98 commented Dec 21, 2018

Terragrunt logs everything to stderr, so terragrunt out <var> should work just fine. If not, please open a separate bug with a way to repro.

@josegonzalez
Copy link

josegonzalez commented Dec 21, 2018

Sorry, I meant that the regex above will not catch all lines from terragrunt - specifically, if you have an un-initialized module, some lines will be missing the prefix. Apologies!

I'd assume that the terraform call also outputs to stderr in some instances, so redirecting all stderr wouldn't be a great solution in those cases.

@brikis98
Copy link
Member

brikis98 commented Dec 21, 2018

Sorry, I meant that the regex above will not catch all lines from terragrunt - specifically, if you have an un-initialized module, some lines will be missing the prefix. Apologies!

Ah, I gotcha, thanks for clarifying.

I'd assume that the terraform call also outputs to stderr in some instances, so redirecting all stderr wouldn't be a great solution in those cases.

Yea, Terraform also logs to stderr. However, command where you are explicitly asking something to be printed such as terraform output write the requested info to stdout.

@voiski
Copy link

voiski commented Jun 7, 2019

use the scenery for this, it will filter out the terragrunt output and you will also win colors =D

--no-color will remove any colors if you need

terragrunt ... | scenery

@tomaszdudek7
Copy link

tomaszdudek7 commented Aug 27, 2019

@voiski

Unfortunately, scenery did not work for me.

It has problems both with terragrunt and terraform 0.12 :(

@scottrbaxter
Copy link

scottrbaxter commented Jan 20, 2020

To add to the bash solution provided by @ezh, I ran into issues with testing syntax during tf .11 --> .12 upgrades (e.g. include { needs to be updated to include = { for tf12), and terragrunt plan-all 2> >(grep -v "\[terragrunt]" >&2) would not show me the error, since it was being redirected. I had to remove this inverse grep stderr redirect to see anything other than:

Command exited with non-zero status 1

I came up with a slightly more explicit regex that helps prune away some of the excess terragrunt stderr output that we don't generally need to see when running every pipeline, however still shows the output to errors such as the one above.

Solution:
2> >(grep -v "^\[terragrunt]\ \[" >&2)

Basically, this grep inversely redirects any lines that start with [terragrunt] [, such as:

[terragrunt] [</path/to/terragrunt.hcl>] 2020/01/20 17:58:40 Running command: terraform --version

We'll still see errors, and other output, such as
1st example:

[terragrunt] 2020/01/20 17:44:53 Error processing module at '/path/to/terragrunt.hcl'. How this module was found: Terragrunt config file found in a subdirectory of path/to. Underlying error: /path/to/terragrunt.hcl:33,15-16: Missing key/value separator; Expected an equals sign ("=") to mark the beginning of the attribute value.

2nd example:

AccessDenied: User: arn:aws:iam::<account-no>:user/<aws-user> is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<account-no>:role/<aws-role>
	status code: 403, request id: <request-id>
[terragrunt] 2020/01/20 18:22:21 Unable to determine underlying exit code, so Terragrunt will exit with error code 1
Command exited with non-zero status 1

I do strongly feel that debugging levels would be very beneficial with terragrunt, and would love to see them implemented eventually, so long as this tool continues to be useful. I do understand that it's getting less necessary alongside terraform with the recent additions found in .12, however there still seems to be enough use unique cases out there that it feels worth the while to implement a way to manage levels of output verbosity - especially considering this stderr redirect (work-around) requires calling bash, when we often use alpine based containers that don't have bash natively added.

@yanivillouz
Copy link

yanivillouz commented Feb 24, 2020

Is there an option that will hide a lot of the extra information that terragrunt outputs? While this information is nice to see, it would be nice to have a clean output that would just show us the raw output from terraform.

redirect stderr to /dev/null and you will have only the terraform outputs in a clean format i.e terragrunt apply-all 2> /dev/null

@scottrbaxter
Copy link

scottrbaxter commented Feb 24, 2020

redirect stderr to /dev/null

@yanivillouz but what if you're troubleshooting an error? also, with the recursive nature of terragrunt *-all, the output is not consistently in order, and often difficult to determine what output belongs to which path/folder. i think some work on better output for testing/troubleshooting (especially developing new patterns) would be hugely useful, here.

@yanivillouz
Copy link

yanivillouz commented Feb 24, 2020

@scottrbaxter
Copy link

scottrbaxter commented Feb 24, 2020

@yanivillouz Your "solution" is not different from the first very response in this issue. Redirecting all stderr for terraform will definitely omit a lot of important output from terraform itself.

Please help avoid excess noise by reading through the entire conversation, if you feel like contributing to issues.

@dmattia
Copy link
Contributor

dmattia commented Apr 9, 2020

On our CI servers, we wanted to display just the plan output when terragrunt ran successfully, but wanted to display all logs if anything went wrong. Here's a gist for a quick node wrapper that achieves our goal for plan calls: https://gist.github.com/dmattia/0d17696bad1dffd90ec7c899e0343955

@plathrop-sfdc
Copy link

plathrop-sfdc commented May 4, 2020

What I'd mostly like to see is deduplicating path components in the output. What I mean is, lines like this:

[terragrunt] [/Users/USER/Source/observability/cortex/v2/live/development/cortex] 2020/05/04 11:27:06 Downloading Terraform configurations from file:///Users/USER/Source/observability/cortex/v2/modules/cortex into /Users/USER/Source/observability/cortex/v2/live/development/cortex/.terragrunt-cache/7W8Zynt-Dn0PhoK3U4XzB9LjtcY/zVzA93tT2UNDH-crbKmeNMrR8Sw

Could be made significantly less verbose, something like:

[terragrunt] [In /Users/USER/Source/observability/cortex/v2] [.../live/development/cortex] 2020/05/04 11:27:06 Downloading Terraform configurations from file://.../modules/cortex into .../live/development/cortex/.terragrunt-cache/7W8Zynt-Dn0PhoK3U4XzB9LjtcY/zVzA93tT2UNDH-crbKmeNMrR8Sw

@angeloskaltsikis
Copy link

angeloskaltsikis commented Jun 26, 2020

I have pushed an update based on @dmattia .
Posting it here in case it helps anyone.

amnk added a commit to amnk/terragrunt that referenced this issue Jan 23, 2021
This moves majority of Terragrunt's output to DEBUG level, and sets
default output mode to ERROR. The idea is that in default run user should
see much less output.

Closes: gruntwork-io#432
amnk added a commit to amnk/terragrunt that referenced this issue Jan 25, 2021
Apart from small fixes, this:
- adds deprecation check for command line arguments, and gives a
  warning if `--terragrunt-debug` is used;
- imports `logrus` under its own name to avoid confusion;

Related: gruntwork-io#432
amnk added a commit to amnk/terragrunt that referenced this issue Feb 2, 2021
This moves majority of Terragrunt's output to DEBUG level, and sets
default output mode to ERROR. The idea is that in default run user should
see much less output.

Closes: gruntwork-io#432
amnk added a commit to amnk/terragrunt that referenced this issue Feb 2, 2021
Apart from small fixes, this:
- adds deprecation check for command line arguments, and gives a
  warning if `--terragrunt-debug` is used;
- imports `logrus` under its own name to avoid confusion;

Related: gruntwork-io#432
yorinasub17 pushed a commit that referenced this issue Feb 2, 2021
* Option for less verbose output

This moves majority of Terragrunt's output to DEBUG level, and sets
default output mode to ERROR. The idea is that in default run user should
see much less output.

Closes: #432

* Updated after review

Apart from small fixes, this:
- adds deprecation check for command line arguments, and gives a
  warning if `--terragrunt-debug` is used;
- imports `logrus` under its own name to avoid confusion;

Related: #432

* Updates after review | 2

Main change - global LogEntry object introduced in `util/logger.go`. It
helps us log properly when we didn't yet parse command line arguments

* Fix tests for `cli/cli_app.go`

P.S. Tests rock!

* Rework global logger

Global logger should be a fallback option, rather a way to go. So
instead of using the same everywhere, we use global if no other logger
is available (for example, if we have not yet parsed arguments).

This change also addresses other PR review comments, like:
- wrong documentation;
- using `logrus.WarnLevel.String()` instead of hardcoded strings;
- updated `isDeprecatedOption()`, with a more clear usage;

* Fix typo in docs section

* Apply `go fmt` properly

* Apply `go fmt` properly | 2

* Fix `TestParseTerragruntOptionsFromArgs` test

* Terragrunt should output all log to stderr

* Set stderr as default output

* Update TestApplySkipTrue()

Add `--terragrunt-log-level info`, so that we could test it.

* Fix tests

This fixes tests that rely on double output of `run_shell`, as well as
brings back `--terragrunt-debug`, but in slightly different form: to
only generate `terragrunt-debug.tfvars.json`

* Remove `--terragrunt-debug` from excluded and fix `\n`

Terragrunt integration tests depend on newlines, and to ensure that
logrus can properly print newlines with `\n` `DisableQuote: true` is
needed [1]. In particular, this fixes
`TestIncludeDirsDependencyConsistencyRegression` and
`TestIncludeDirsStrict`.

[1] - sirupsen/logrus#608

* Bring `--terragrunt-debug` tests back
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.