GCP Data Pipeline with Terragrunt

A complete GCP data pipeline implementation using Terragrunt for infrastructure as code. This pipeline demonstrates a production-ready data engineering architecture with batch and streaming processing, following a modular structure with numbered directories.

Architecture

• Source Project: Read-only BigQuery access (configured via environment variables)
• Processing Project: All compute resources and data writes (configured via environment variables)
• Components: Dataproc cluster, BigQuery, Pub/Sub, GCS, Data Catalog, Service Accounts

Security

Critical: No PII, secrets, or sensitive data in code. All sensitive values are stored in .env file (git-ignored).

1. Copy .env.example to .env
2. Fill in all required values in .env
3. Never commit .env to version control

Prerequisites

• Google Cloud SDK (gcloud) installed and configured
• Terraform >= 1.0
• Terragrunt >= 0.45
• Python 3.8+ (for scripts)
• Access to both GCP projects with appropriate permissions

Project Structure

gcp-data-pipe/
├── terragrunt.hcl (root)
├── README.md
├── .gitignore
├── .env.example
├── .env (user-created, git-ignored)
├── 01-iam/          # Service accounts and IAM bindings
├── 02-storage/      # GCS buckets
├── 03-bigquery/     # BigQuery datasets and tables
├── 04-pubsub/       # Pub/Sub topics and subscriptions
├── 05-dataproc/     # Dataproc cluster
├── 06-datacatalog/  # Data Catalog entry groups
├── 07-orchestration/ # Cloud Functions and Scheduler (optional)
└── scripts/         # PySpark jobs and utility scripts

Setup

1. Clone and navigate to the project:

   git clone https://github.com/NeverTheSame/gcp-data-pipe.git
   cd gcp-data-pipe

2. Create .env file:

   cp .env.example .env
   # Edit .env and fill in all required values

3. Set up Terraform backend:

   • Ensure the GCS bucket for Terraform state exists
   • Update TF_STATE_BUCKET and TF_BACKEND_CREDENTIALS_PATH in .env

4. Deploy infrastructure (in order):

   # Step 1: IAM
   cd 01-iam
   terragrunt apply
   cd ..
   
   # Step 2: Storage
   cd 02-storage
   terragrunt apply
   cd ..
   
   # Continue with other modules...

Data Sources

• Primary: Real logging data from dev-hfs-fat-34e0 BigQuery tables
• Fallback: Synthetic data when production data is unavailable

Usage

Quick Start

1. Set up environment:

   cp .env.example .env
   # Edit .env with your values

2. Deploy infrastructure (in order):

   cd 01-iam && terragrunt apply && cd ..
   cd 02-storage && terragrunt apply && cd ..
   cd 03-bigquery && terragrunt apply && cd ..
   cd 04-pubsub && terragrunt apply && cd ..
   cd 05-dataproc && terragrunt apply && cd ..
   cd 06-datacatalog && terragrunt apply && cd ..

3. Upload and run batch job:

   ./scripts/upload.sh
   ./scripts/submit_job.sh

4. Run validation:

   ./scripts/validate.sh

Running Scripts

• Batch ETL: ./scripts/submit_job.sh
• Streaming Producer: python3 scripts/producer.py
• Data Catalog Tagging: python3 scripts/tag_tables.py
• End-to-End Test: ./scripts/run_all.sh

Validation

Run validation scripts to verify:

• Read-only access to source project
• Write permissions in processing project
• End-to-end pipeline execution

./scripts/validate.sh

See VALIDATION.md for detailed validation steps and troubleshooting.

The PySpark job will automatically:

1. Try to read from configured source tables
2. Fall back to synthetic data if source tables don't exist
3. Document which data source was used

Troubleshooting

See VALIDATION.md for common issues and solutions.

Common Commands

# Check Dataproc cluster status
# Replace ${GCP_PROCESSING_PROJECT_ID} with your project ID from .env
gcloud dataproc clusters describe ${DATAPROC_CLUSTER_NAME} \
  --project=${GCP_PROCESSING_PROJECT_ID} \
  --region=${GCP_REGION}

# List Dataproc jobs
gcloud dataproc jobs list \
  --project=${GCP_PROCESSING_PROJECT_ID} \
  --region=${GCP_REGION} \
  --cluster=${DATAPROC_CLUSTER_NAME}

# Query processed data
# Replace with your project ID and dataset from .env
bq query --project_id=${GCP_PROCESSING_PROJECT_ID} \
  "SELECT * FROM \`${GCP_PROCESSING_PROJECT_ID}.${BQ_PROCESSED_DATASET}.symbol_metrics\` LIMIT 10"