Warning This package is a pre-released of the official apache-airflow-providers-google package. All of these operators will be integrated to the official package, soon.
This repository provides an Apache Airflow provider based on BigQuery Reservation API.
BigQueryReservationCreateOperator
: Buy BigQuery slots (commitments) and assign them to a GCP project (reserve and assign).BigQueryReservationDeleteOperator
: Delete BigQuery commitments and remove associated ressources (rservation and assignment).BigQueryBiEngineReservationCreateOperator
: Create or Update a BI engine reservation.BigQueryBiEngineReservationDeleteOperator
: Delete or Update a BI engine reservation.
You could find DAG samples here.
- A Google Cloud connection has to be defined.
By default, all hooks and operators use
google_cloud_default
. - This connection requires the following roles on the Google Cloud project(s) used in these operators:
- BigQuery Resource Admin
- BigQuery Job User - Required for
BigQueryReservationCreateOperator
because of the reservation attachment check.
Defining a new dedicated connection and custom GCP role could be good practices to respect the principle of least privilege.
pip install --user airflow-provider-bigquery-reservation