[CT-2767] [Feature] Use --quiet by default for dbt list --output json

[dbeatty10]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

It is often useful to pipe or redirect JSON output of dbt list for further downstream processing with a tool like the Python built-in json.tool or the jq swiss army knife.

It would be nice if the syntax were as simple as this to pretty-print the JSON output:

dbt list --output json | python -m json.tool --json-lines

or this to extract the name keys:

dbt list --output json | jq ".name"

But currently, the non-error log output needs to be suppressed like this:

dbt --quiet list --output json | python -m json.tool --json-lines
dbt -q list --output json | jq ".name"

See #7987 for an example.

Relevant background

The guide for upgrading to v1.5 says the following:

The dbt list command will now include INFO level logs by default. Previously, the list command (and only the list command) had WARN-level stdout logging, to support piping its results to jq, a file, or another process. To achieve that goal, you can use either of the following parameters:

• dbt --log-level warn list (recommended; equivalent to previous default)
• dbt --quiet list (suppresses all logging less than ERROR level, except for "printed" messages and list output)

Describe alternatives you've considered

It's certainly possible to continue to require usage of the QUIET config, it's just harder for folks to discover.

The problem is that the default option gives output that is un-parsable by any downstream tooling 😢

Who will this benefit?

Anyone that want to do any JSON post-processing in a terminal.

Are you interested in contributing this feature?

No response

Anything else?

If we were to adopt this enhancement request, would we also need/want to add a --no-quiet option to remove the suppression of non-error output? I can't think of a particular use-case off-hand, but it wouldn't surprise me if one exists.

[jtcohen6]

Thanks @dbeatty10! I agree this would be an improvement, versus requiring all users to read the note in the v1.5 migration guide.

I think the implementation here could be fairly tricky, because our event manager (logging system) is initialized before the list task is initialized. But we might be able to have some sort of special-case behavior if (and only if) it's --output json. I don't want to see us going back to the exceptionally special behavior we had for list in previous versions, which achieved the right UX but was inflexible & difficult to reason about.

Thoughts:

• There's definitely a --no-quiet flag!
• --quiet is sugarier syntax; technically --log-level warn is a more precise parallel with the previous (pre-1.5) behavior
• Should we also switch --log-format json on by default? If so, we'd need to change the way that list outputs its results. Currently we use print if not JSON, real log events if not — a finicky implementation IMO:

dbt-core/core/dbt/task/list.py

Lines 145 to 149 in 5d93780

	if get_flags().LOG_FORMAT == "json":
	fire_event(ListCmdOut(msg=result))
	else:
	# Cleaner to leave as print than to mutate the logger not to print timestamps.
	print(result)

[graciegoheen]

Notes from refinement:

• warning about complexity is that when we're running the preflight method we haven't necessarily created the list command, and this behavior (we would know what command was going to be run via the flags, maybe reaching down into the commmand context)
• there's extra fields that show up in JSON, so would need to go this route to get all of the attributes (normal list output is simplified / "human readable")

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.