An end-to-end data pipeline which extracts divvy bikeshare data from web loads it into data lake and datawarehouse transforms it using dbt and finally , a dashboard to visualize the data using looker studio, the pipeline is orchestrated using prefect

Modeling tool like DBT to use SQL Alchemy core with a DataFrame interface like

DataForge helps data teams write functional transformation pipelines by leveraging software engineering principals

A modern data platform implemented on Azure Synapse Analytics using ELT Framework - https://github.com/bennyaustin/elt-framework. Data platform infrastructure provisioned using https://github.com/bennyaustin/iac-synapse-dataplatform

🛸 This project showcases an Extract, Load, Transform (ELT) pipeline built with Python, Apache Spark, Delta Lake, and Docker. The objective of the project is to scrape UFO sighting data from NUFORC and process it through the Medallion architecture to create a star schema in the Gold layer that is ready for analysis.

Public DBT instance to aid in data transformation for analytics purposes

Data engineering projects

Irish Property Price Register transformed into a data warehouse via an EtLT pipeline.

🍺 A data engineering project showcasing an ELT pipeline using modern technologies such as Delta-rs, and Apache Airflow.

SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline

Builded an ETL pipeline using Python, Pandas, Python dictionary methods and regular expressions to ETL data. It involves extracting data from multiple sources, cleaning and transforming the data using Jupyter Notebook with pandas, numpy, and datetime packages, and loading the cleaned data into a relational database using pgAdmin

Data Engineering project which involves ETL using PostgreSQL and Python

BigQuery data pipeline with dbt, Spark, Docker, Airflow, Terraform, GCP

Example ELT data pipeline project using dbt

An ELT pipeline built for the pNEUMA open dataset of naturalistic trajectories of half a million vehicles collected by a swarm of drones in a congested downtown area of Athens, Greece.

ELT Project

This is an ELT data pipeline setup to track the activities of an e-commerce website based on orders, reviews, deliveries and shipment date. This project utilized technologies like Airflow, AWS RDS-Postgres, Python etc.

ELT Project that ultilizes Docker and PostgreSQL for a simple ELT process