prefect-dbt Add back logging of dbt commands #13853
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #13699
In 0.5.0, the functionality was changed to call dbt in Python with the output placed in an artifact.
As per the issue above, for long running dbt commands, it is important to see the status of the dbt command in Prefect while it is running. This logging was removed in 0.5.0 and so this PR adds back this functionality as default.
Initially I attempted to solve this by setting PREFECT_LOGGING_EXTRA_LOGGERS and PREFECT_LOGGING_LOG_PRINTS, but these do not seem to work for this case.
This PR therefore uses the callback functionality to reintroduce this logging: https://docs.getdbt.com/reference/programmatic-invocations#registering-callbacks
I've parameterised this (default: True), so it can be turned off if not needed. This logging was on by default for all versions before 0.5.0, therefore I have implemented it on by default going forward.
There are also some inaccuracies in the commentary, which I have addressed for this function.
Checklist
<link to issue>
"maintenance
,fix
,feature
,enhancement
,docs
.For documentation changes:
mint.json
for files that are removed or renamed.For new functions or classes in the Python SDK:
docs/mint.json
navigation.