Example end to end data engineering project.
-
Updated
Dec 8, 2022 - Python
Example end to end data engineering project.
Replicate data from MySQL, Postgres and MongoDB to ClickHouse
Ecommerce Realtime Data Pipeline (Data Modeling, Workflow Orchestration, Change Data Capture, Analytical Database and Dashboarding)
Repo for CDC with debezium blog post
Built a real-time streaming pipeline to extract stock data, using Apache Nifi, Debezium, Kafka, and Spark Streaming. Loaded the transformed data into Glue database and created real-time dashboards using Power BI and Tableau with Athena. The pipeline is orchestrated using Airflow.
Stream data between two databases . Supports both ddl and dml statements. Built on top of kafka and debezium in python
Guardian for your Kafka Connect connectors. It check status of connectors and tasks and restart if they are failed
Outbox pattern using Debezium and Protobuf serialization
Data Pipeline for CDC data from MySQL DB to Amazon S3 through Amazon MSK using Amazon MSK Connect (Debezium).
Change Data Capture (CDC) refers to sourcing database change events from a source database. While there are commercial solutions available in the market, Debezium is available as an open-source option. In this blog post, I am going to show you how to install the Debezium MySQL Connector on Ubuntu machines using Google VM instances.
Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming and MSK Connect (Debezium)
Data Pipeline for CDC data from MySQL DB to Amazon S3 through Amazon MSK Serverless using Amazon MSK Connect (Debezium).
This repository provides a robust real-time sales data pipeline powered by Apache Kafka, Debezium, Apache Spark, MySQL, and Metabase. It streamlines the seamless ingestion, processing, storage, and visualization of sales data streams, enabling efficient real-time analytics and insights.
Add a description, image, and links to the debezium topic page so that developers can more easily learn about it.
To associate your repository with the debezium topic, visit your repo's landing page and select "manage topics."