Add BigQuery reservation configuration#2096
Conversation
|
/gcbrun |
|
@kolina there were some |
|
/gcbrun |
…urations, protos, and related code Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
… config assertions Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
|
/gcbrun |
|
|
||
| // Optional. The default BigQuery reservation to use for execution. | ||
| // If unset, default BigQuery behavior applies. | ||
| string default_reservation = 14; |
There was a problem hiding this comment.
I'd add a comment that it'll be supported by the Dataform CLI only for now (support in GCP Dataform will require about to the GCP Dataform executor to use this value), same for action fields below
There was a problem hiding this comment.
😔
So cloud Dataform executor has its own version of cli/api/dbadapters/bigquery.ts?
Then all the action reservation attributes will not impact anything in GCP Dataform.
How often does it catch up?
There was a problem hiding this comment.
Yeah, the GCP Dataform executor is different.
We were planning to implement it later this year (Q2-Q3 timeframe), we may do it sooner taking your contribution.
|
/gcbrun |
|
|
||
| // Optional. The default BigQuery reservation to use for execution. | ||
| // If unset, default BigQuery behavior applies. | ||
| string default_reservation = 14; |
There was a problem hiding this comment.
Yeah, the GCP Dataform executor is different.
We were planning to implement it later this year (Q2-Q3 timeframe), we may do it sooner taking your contribution.
2 participants
This pull request introduces support for specifying BigQuery reservations at both the project and action level, allowing more granular control over query resource allocation. The changes span CLI flags, project configuration, action builders, and test coverage to ensure correct application and precedence of reservation settings.
BigQuery reservation support:
--default-reservationCLI flag and correspondingdefaultReservationproject config option, enabling users to set a default BigQuery reservation for all actions in a project.IBigQueryExecutionOptionsandBigQueryDbAdapterto accept and propagate areservationparameter, ensuring reservation information is passed through to BigQuery job execution.Action-level reservation configuration:
Assertion,Table,IncrementalTable,Operation,View) to support an optionalreservationconfig property, storing it in the action descriptor for runtime precedence over project-level reservations.Test coverage and validation:
Project configuration precedence:
reservationoverrides the project-leveldefaultReservationat runtime, with tests confirming the correct behavior.These changes provide users with flexible reservation management for Dataform workload.
Related to #1981