Installation

• Create a new bq-autoload Google Cloud Storage bucket

  $> gsutil mb -c regional -l europe-west1 "gs://bq-autoload"

• Create a new Staging BigQuery dataset

  $> bq mk --dataset "Staging"

• Clone and deploy this repository as a cloud function triggered by changes on this GCS bucket (do not forget to replace the project id)

  $> git clone "https://github.com/tfabien/bigquery-autoload/"              \
     && cd "bigquery-autoload"                                              \
     && gcloud functions deploy "bigquery-autoload"                         \
                      --trigger-bucket "bq-autoload"                        \
                      --set-env-vars "PROJECT_ID={{YOUR_GCP_PROJECT_ID}}"   \
                      --runtime "nodejs10"                                  \
                      --memory "128MB"

Any file you upload to the bq_autoload GCS bucket will now automatically be loaded into a BigQuery table within seconds.