ETL pipeline with PySpark on Dataproc for data lake on Google Cloud Storage

Get started with Prefect by scheduling your Prefect flows with GitHub Actions

Checking the scalability of a data pipeline involving MySQL, Spark and Machine Learning Models using Latency.

This repository contains code for comparing the performance of three different ELT (Extract, Load, Transform) methods on CSV files of different sizes. The three methods are implemented in Python using different approaches and libraries, and their execution times are compared and plotted for analysis.

A custom Airbyte connector to fetch football data from the Football-Data.org API. It allows users to retrieve match results, league tables, and player statistics for specific leagues, making it a versatile tool for football data analysis.

Batch/stream ETL pipeline of NOAA GLM dataset, using Python frameworks: Dagster, PySpark and Parquet storage.

Airflow DAG tutorial with docker compose local setup

Pipeline de dados automatizado para extração e armazenamento de previsões meteorológicas para o setor de turismo.

Latency Estimation for Neural Network Architecture

ETL pipeline with AWS Redshift orchestrated with Airflow

Create Data Pipeline with Apache Airflow for Sparkify Datasets.

Data-Warehouse-using-AWS-S3-and-Redshift

Data pipeline to gather data from chess website APIs using Airflow.

An end-to-end data pipeline deployed on GCP that extracts cryptocurrency data for analytics.

The mini project for the course Database Technologies. The task is to take in data via a pipeline built using spark-streaming and kafka, and store the processed data into a SQLite database for further manipulation

Deployable AWS data platform to process powerlifting data extracted from openpowerlifting.org.

A Python wrapper for making requests to the NYT Entity Service API

Python module that adds unix-like pipe operation and adapts common python functions

Command line interface for the FAIR Data Pipeline

etl pipeline for turkish football events