An end-to-end data pipeline which extracts divvy bikeshare data from web loads it into data lake and datawarehouse transforms it using dbt and finally , a dashboard to visualize the data using looker studio, the pipeline is orchestrated using prefect

A Python package to enrich Google Cloud Data Catalog Fileset Entries with tags.

Interacting with public cloud object storage

Batch ETL pipeline project on GCP to load and transform daily flight data using Spark to update tables in BigQuery. The pipeline is automated using Airflow.

Airflow script that ingest data from API and upload it to gcs bucket

Ingesting NYC Yellow Taxi Data to GCP using Airflow running on Docker

Creating a robust and scalable data pipeline on Google Cloud Platform (GCP) to monitor and analyze stock performance. Leveraging the power of GCP's data processing and storage services, a comprehensive solution has been built to efficiently collect, process, and visualize stock data.

This is in order to add additional stocks data information using dbt.

Simplified blueprints for building data pipelines with Google Cloud Storage (GCS).

Util functions to organize train/test data

Simple python scripts to upload local files into GCP's bucket containerized with docker.

This GitHub repository serves as a comprehensive platform for managing and showcasing my data engineering projects and assessments throughout my final semester at Alt School Africa. Designed to foster collaboration, organization, and continuous improvement, this repository is the backbone of my academic journey in data engineering.

This project aims to migrate data from MongoDB to Google Cloud Storage (GCS) and BigQuery automatically. It enables businesses to easily transfer and analyze the data in the cloud, improving data and cost management.

Lambda function for 'webapp' to save files in GCP bucket and logs in DynamoDB tables when SNS topic triggers

Analysis of NYC's citibike data. Technologies: Python , Prefect, dbt, Terraform , Looker data studio

An ETL pipeline to move an uploaded flat file ffrom GCS, mask PII, store Big Query, and Create a report in Looker.