Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-692] Enrich the adapter response to analyze executed BigQuery jobs in detail #194

Closed
yu-iskw opened this issue May 27, 2022 · 1 comment · Fixed by #271
Closed

[CT-692] Enrich the adapter response to analyze executed BigQuery jobs in detail #194

yu-iskw opened this issue May 27, 2022 · 1 comment · Fixed by #271
Labels
bytes_processed Issues related to cost tracking in BigQuery enhancement New feature or request good_first_issue Good for newcomers

Comments

@yu-iskw
Copy link
Contributor

yu-iskw commented May 27, 2022

Describe the feature

We are able to get bytes_processed with adaptor responses.
We put total_bytes_processed to the adaptor response. So, we are able to estimate cost of a query with results.adapter_response["bytes_processed"] in run_results.json.
When a BigQuery user takes advantage of the flat fare plan, consumed slots milliseconds is more important to improve resource allocation. So, it would be great to enrich the adapter resource by adding slot_millis and so on. By doing that, we can analyze executed BigQuery queries by dbt in detail.

https://github.com/dbt-labs/dbt-bigquery/blob/1.1.latest/dbt/adapters/bigquery/connections.py#L86-L88

Describe alternatives you've considered

I am not sure if we are able to get something like slot_mills of the QueryJob class from dbt artifacts.

Additional context

Who will this benefit?

People who want to analyze executed queries by dbt in detail

Are you interested in contributing this feature?

I would love to implemented the feature.

@yu-iskw yu-iskw added enhancement New feature or request triage labels May 27, 2022
@github-actions github-actions bot changed the title Enrich the adapter response to analyze executed BigQuery jobs [CT-692] Enrich the adapter response to analyze executed BigQuery jobs May 27, 2022
@yu-iskw yu-iskw changed the title [CT-692] Enrich the adapter response to analyze executed BigQuery jobs [CT-692] Enrich the adapter response to analyze executed BigQuery jobs in detail May 27, 2022
@jtcohen6
Copy link
Contributor

jtcohen6 commented Jun 1, 2022

Makes a lot of sense to me @yu-iskw! Thanks for clearly laying out the issue, and opening the accompanying PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bytes_processed Issues related to cost tracking in BigQuery enhancement New feature or request good_first_issue Good for newcomers
Projects
None yet
3 participants