Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prefect-dbt Add back logging of dbt commands #13853

Merged
merged 6 commits into from
Jun 10, 2024
Merged

Conversation

padbk
Copy link
Contributor

@padbk padbk commented Jun 7, 2024

Closes #13699

In 0.5.0, the functionality was changed to call dbt in Python with the output placed in an artifact.

As per the issue above, for long running dbt commands, it is important to see the status of the dbt command in Prefect while it is running. This logging was removed in 0.5.0 and so this PR adds back this functionality as default.

Initially I attempted to solve this by setting PREFECT_LOGGING_EXTRA_LOGGERS and PREFECT_LOGGING_LOG_PRINTS, but these do not seem to work for this case.

This PR therefore uses the callback functionality to reintroduce this logging: https://docs.getdbt.com/reference/programmatic-invocations#registering-callbacks

I've parameterised this (default: True), so it can be turned off if not needed. This logging was on by default for all versions before 0.5.0, therefore I have implemented it on by default going forward.

There are also some inaccuracies in the commentary, which I have addressed for this function.

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • This pull request includes a label categorizing the change e.g. maintenance, fix, feature, enhancement, docs.

For documentation changes:

  • This pull request includes redirect settings in mint.json for files that are removed or renamed.

For new functions or classes in the Python SDK:

  • This pull request includes helpful docstrings.
  • If a new Python file was added, this pull request contains a stub page in the Python SDK docs and an entry in docs/mint.json navigation.

@padbk
Copy link
Contributor Author

padbk commented Jun 10, 2024

Please let me know what I need to do to help get this merged. Thanks,

Copy link
Collaborator

@chrisguidry chrisguidry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks for the contribution, @padbk!

@chrisguidry chrisguidry merged commit 5a93678 into PrefectHQ:main Jun 10, 2024
12 checks passed
@padbk padbk deleted the patch-1 branch June 10, 2024 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

prefect_dbt: Bring back logging and stream output of dbt cli comamnds in trigger_dbt_cli_command function
2 participants