library for batch-oriented complex data integration pipelines

ETL Process for Currency Quotes Data" project is a complete solution dedicated to extracting, transforming and loading (ETL) currency quote data. This project uses several advanced techniques and architectures to ensure the efficiency and robustness of the ETL process.

No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents

Generative AI & Recommendation Engine --- Firat University / Faculty of Technology / Software Engineering / Final Project

A framework for writing Unstract Tools/Apps

Airflow DAGs for the Stellar ETL project

End-to-end ETL pipeline in the Microsoft Azure cloud.

Expressive iteration in Python: fluent, typed, lazy, and concurrent

GFW Data API

One ETL tool to rule them all

Simple HWM Store backend

Horizon HWM Store for onETL

Created a ETL pipeline for Pokemon using scraping for extraction and sqlite for loading

This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.

Personal Spotify Wrapped using Airflow, Django and Docker

Developing ETL pipeline to process FDA Data using Apache Airflow, AWS S3, Snowflake, Python pandas, Docker and Draw meaningful insights through visualization with Tableau.

This project is designed to handle data read/write to any database and works well as an ETL engine

Final Assignment Submission: ETL and Data Pipelines with Shell, Airflow and Kafka

This repository contains the NYC Taxi Data Engineering Pipeline project, which aims to build a comprehensive data engineering pipeline using NYC taxi data from the years 2022 and 2023. The pipeline involves extracting, transforming and loading (ETL) data into a Snowflake database, followed by creating a dashboard for visualisation.