High Performance Tensorflow Data Pipeline with State of Art Augmentations and low level optimizations.

Simple stream processing pipeline

kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.

Материалы для курса Введение в Data Engineering: дата пайплайны

Modeling tool like DBT to use SQL Alchemy core with a DataFrame interface like

Simple Airflow on Kubernetes (GKE)

This course is designed to provide learners with the fundamental skills needed for data engineering using Python. The objective is to introduce anyone interested in the topic to Python's data engineering-related features.

This is an ETL project - extracting data from an ecommerce transactional database on RDS, transforming the data using AWS glue job, and loading it to a Redshift data warehouse, and connected it to Tableau for BI

An ETL data pipeline that extracts data from source and loads it to destination, automated using mage.ai

AWS 데이터파이프라인 개발과 음악추천 챗봇

This is a project which demonstrates creation of a data pipeline by scraping data using twitter API and creating a data delivery stream using Kinesis Firehose for ingesting data to Amazon S3.

GFW ETL for raster tiles

DBT and clickhouse test project with dagster

An Automated data pipeline using "Apache Airflow" performing "ETL" on RAW data using "Pandas" library then stage data into "PostgreSQL" then process it distributed cluster and parallelly using "Spark" and loaded final useful data into "ElasticSearch" NoSQL DB warehouse

Generate new places to watch data when GLAD is updated

A data pipeline to daily pull public transport data from the opentransportdata.swiss portal. This pipeline has three tasks, pull the right data from opentransportdata.swiss, push the data to s3 for storage, and transform and load the transformed data to a database. Hopefully this repository helps people explain ETL / Batch data pipeline.

Dagster Tutorial to get you started with Dagster as an absolute Beginner. The tutorial covers various topics like Dagster Installation, Dagster Asset, Dagster Job, Dagster Scheduler, Dagster Ops, and more. It is completely free on YouTube and is beginner-friendly without any prerequisites.

A data pipeline to analyze real time cryptocurrency price