bigquery-to-postgres-etl

ETL --> (Extract - Load)

[ Source --> BigQuery, Destination --> Postgres]

Project Overview

Description: Personal project (BQ to Postgres data ingestion)

Source: GCP Big Query

Apache Airflow (Orchestration Tool) : Not Involved in this project*

Destination: Postgresql

Language: python

Main Tools Involved:

1. Python (IDE - Visual Studio Code)
2. Google Cloud Platform account (For extracting data from BigQuery)
3. Docker (Preferred)

Workflow

1. Extract data from BigQuery via query
2. Save queried output into dataframe and output data as csv format
3. Ingest data into Postgres DB with new file name

Final Outcome

Extra Information

1. Docker Compose tutorial (to run postgresql)

--> DE Zoomcamp 1.2.5 - Running Postgres and pgAdmin with Docker-Compose