gcp-bigquery

Data sync via CDC from GCP Cloud SQL to Big Query using Datastream.

Make sure to login with gcloud:

gcloud auth application-default login

First cd into the infra directory.

Create the .auto.tfvars parameter:

project_id        = "<project_id>"
region            = "us-central1"
database_tier     = "db-g1-small"
database_username = "dbadmin"
database_password = "p4ssw0rd"

Create the database:

terraform init
terraform apply -auto-approve

Connect to the database and execute execute the SQL commands in the database.sql file.

This database already has automated backups and point-in-time recovery enabled.

Now in the GCP console create a stream in Datastream that sends the changes in the database to BigQuery.

Data should be available in the BigQuery dashboard for queries.