Implementing best practices for PySpark ETL jobs and applications.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

This repo will guide you step-by-step method to create star schema dimensional model.

An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.

Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE

Built a Data Pipeline for a Retail store using AWS services that collects data from its transactional database (OLTP) in Snowflake and transforms the raw data (ETL process) using Apache spark to meet business requirements and also enables Data Analyst create Data Visualization using Superset. Airflow is used to orchestrate the pipeline

Introduction to the data pipeline management with Airflow. Airflow schedule and maintain numerous ETL processes running on a large scale Enterprise Data Warehouse.

Sentiment Analysis of Tweets Using ETL process and Elastic Search

Example project and best practices for Python-based Spark ETL jobs and applications.

An ETL pipeline where data is captured from REST API (Remotive, Adzuna & GitHub) and RSS feeds (StackOverflow). The data collected from the API is stored on local disk. The files are preprocessed and ETL jobs are written in spark and scheduled in Prefect to run every week. Transformed data is moved to PostgreSQL.

python 3.5 package for ETL jobs

Event-Driven Python on AWS #CloudGuruChallenge

This project involves using the Reddit API to extract data, processing it using EC2 instances, and storing the output in CSV format within an AWS S3 bucket, with Airflow managing the overall workflow orchestration.

Code for unofficial API for the brazillian stocks data website called Fundamentus. Uses requests and bs4 for scraping

Data pipeline using S3, Glue, Athena, Lambda and Quicksight to analyze dataset of YouTube

Utilities for declarative specification of data download pipelines for ETL jobs.

Apache Airflow installation with Docker 🌬️

ETL (Extract, Transform, Load) job using PySpark - submodule

This is a command line application to demonstrate a sample ETL pipeline in python. It takes a PostgreSQL dataset that is provided by https://raw.githubusercontent.com/cdvx/etl-python/movies-sql/movielens.sql and transfers the data to MongoDB.

A data pipeline with Docker to perform Sentiment Analysis on tweets and post it on a slack channel via a bot