Data Engineering examples covering Airflow and Mage for workflows; dbt for BigQuery, Redshift, ClickHouse; Spark and Kafka for Batch/Streaming Processing
-
Updated
May 20, 2024 - Python
Data Engineering examples covering Airflow and Mage for workflows; dbt for BigQuery, Redshift, ClickHouse; Spark and Kafka for Batch/Streaming Processing
A simple demo showing how to use Ably and fastAPI to route messages into Kafka for stream processing
Current 2022 Confluent Keynote Demo covering Stream Designer, Stream Catalog, and Stream Sharing.
Interactive ksqlDB command line client with autocompletion and syntax highlighting written in Python
For recreational use. Just a playground of Kafka+Spark+MQTT+KSQLDB+others
This repository includes data engineering projects using Apache Airflow. I hope to add more projects using different technologies soon!
An app to keep track of Youtube videos and sends the notification to a Telegram bot to inform you if anyone comments on those
Real-time Coinbase market data streaming pipeline with visualizations. Much appreciation to DataTalks.Club Data Engineering Zoom Camp: https://github.com/DataTalksClub/data-engineering-zoomcamp
Kafka Connect and kSQLDB with Oracle
This repository contains a KSQLDB setup connection that streams the average ETH Gas Estimate using the Ethereum Gas Estimate API as source data.
Kubernetes demo
Building a streaming Kafka application to push live notifications for updates to views, likes, favorites and comments on Youtube videos.
Real time fraud analysis using Kafka Streams
Add a description, image, and links to the ksqldb topic page so that developers can more easily learn about it.
To associate your repository with the ksqldb topic, visit your repo's landing page and select "manage topics."