Skip to content


Repository files navigation

Actions Status Actions Status Version Plugin Status

BigQuery Buildkite Plugin

This Buildkite plugin can be used to deploy tables/views schemas to BigQuery

A Buildkite plugin for deploying tables/views schemas to BigQuery.

Using the plugin

If the version number is not provided then the most recent version of the plugin will be used. Do not use version number as master or any branch names.


  - plugins:
      - wayfair-incubator/bigquery#v1.2.0:
          gcp_project: gcp-us-project
          dataset_schema_directory: schemas/gcp-us-project/dataset



gcp_project (required, string)

The full name of the GCP project you want to deploy.

Example: gcp-us-project

dataset_schema_directory (required, string)

The directory in your repository where are you storing the schemas for your tables and views.

Example: gcp-us-project/dataset_name

prod_build_branch (optional, string)

Default branch for the repo.

Example: main

Default: master

execute_only_changed_files (optional, boolean)

Whether to execute just the files that changed in a directory or not.

Example: true

Default: true

fail_pipeline_on_first_exception (optional, boolean)

Whether to fail pipeline as soon as there is one failure.

Example: true

Default: true


This plugin expects GCP_SERVICE_ACCOUNT is placed as environment variable. Make sure to store it securely!

  gcp_service_account: '{"email": ""}'



The following pipeline will deploy all the schemas to the gcp-us-project living under directory/project/


This plugin uses GBQ to deploy to Google BigQuery. GBQ now supports specifying partitions with the schema as well.

To leverage this you need to nest your JSON table schema in a dictionary. An example for the same is given below. Library supports Time and Range based partitioning along with Clustering.

All the configuration options can be found here.

  "partition": {
    "type": "range",
    "definition": {
      "field": "ID",
      "range": {
        "start": 1,
        "end": 100000,
        "interval": 10
  "clustering": [
  "schema": [
      "name": "ID",
      "type": "INTEGER",
      "mode": "REQUIRED"


See the Contributing Guide for additional information.

To execute tests locally (requires that docker and docker-compose are installed):

docker-compose run test


This plugin was originally written by Jash Parekh for Wayfair.