Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2950] [Applied State] Fire ResourceReport Event at End of Run #8342

Closed
Tracked by #8316
peterallenwebb opened this issue Aug 8, 2023 · 0 comments · Fixed by #8671
Closed
Tracked by #8316

[CT-2950] [Applied State] Fire ResourceReport Event at End of Run #8342

peterallenwebb opened this issue Aug 8, 2023 · 0 comments · Fixed by #8671
Assignees

Comments

@peterallenwebb
Copy link
Contributor

peterallenwebb commented Aug 8, 2023

To make it easier to track dbt's resource usage, an event summarizing resource consumption should be fired at the end of an invocation, and include important resource usage statistics.

At a minimum, this should include the peak memory usage in bytes, as it would appear in top or the Windows Task Manager. While we are implementing this event, however, we should look at including CPU usage (with user/kernel breakdown), and perhaps some block I/O stats.

Because Python does not have a standard mechanism for collecting this information, we will have to consider a library like pstools or limit support to specific OSes (for example, by using the /proc filesystem on linux).

Acceptance Criteria

  • After executing a command, dbt fires an INFO-level event called ResourceReport.
  • The ResourceReport event contains a property called process_mem_max_rss, a 64-bit integer representing the peak memory usage during the command execution, in bytes.
@github-actions github-actions bot changed the title [Applied State] Fire ResourceReport Event at End of Run [CT-2950] [Applied State] Fire ResourceReport Event at End of Run Aug 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants