Deploy Kafka pipelines to Kubernetes
-
Updated
Jun 6, 2024 - Python
Deploy Kafka pipelines to Kubernetes
python-prometheus-export
Replicate data from MySQL, Postgres and MongoDB to ClickHouse
This repository includes data engineering projects using Apache Airflow. I hope to add more projects using different technologies soon!
This repository serves as a comprehensive resource for architectural templates and examples for modern software integrations. Specifically designed using Apache Kafka.
Example pipeline to stream the data changes from RDBMS to Apache Iceberg tables
Explore Apache Kafka data pipelines in Kubernetes.
A streaming data pipeline uses Kafka as the backbone and Flink for data processing and transformations. Kafka Connect is used for writing the streams to S3 compatible blob stores and Redis (low latency KV store for real-time ML inference). Spark is used for the batch job to backfill the ml feature data.
Django with Kafka, Debezium, and Faust for Email Sending using Change Data Capture
Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming using Amazon MSK and MSK Connect (Debezium)
Data Pipeline for CDC data from MySQL DB to Amazon S3 through Amazon MSK Serverless using Amazon MSK Connect (Debezium).
Data Pipeline for CDC data from MySQL DB to Amazon S3 through Amazon MSK using Amazon MSK Connect (Debezium).
Data Pipeline for CDC data from MySQL DB to Amazon S3 through Amazon MSK Serverless using Amazon MSK Connect (Debezium).
Guardian for your Kafka Connect connectors. It check status of connectors and tasks and restart if they are failed
Repositório destinado a estudos referente apache-kafka
Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming and MSK Connect (Debezium)
Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming and MSK Connect (Debezium)
Add a description, image, and links to the kafka-connect topic page so that developers can more easily learn about it.
To associate your repository with the kafka-connect topic, visit your repo's landing page and select "manage topics."