Stream Processing of website click data using Kafka and monitored and visualised using Prometheus and Grafana

Deploy Kafka pipelines to Kubernetes

Real-Time Sentiment Analysis on Twitter Streams is a web application that categorizes tweets into sentiments like Negative, Positive, Neutral, or Irrelevant. Built using Apache Kafka , Spark and PySpark ML models, it offers real-time analysis capabilities.

Data Engineering examples covering Airflow and Mage for workflows; dbt for BigQuery, Redshift, ClickHouse; Spark and Kafka for Batch/Streaming Processing

This Bash script automates the setup and execution of a data processing pipeline using Apache Kafka and Python scripts, ensuring fault tolerance and streamlined management of Kafka-based data pipelines.

Explore Apache Kafka data pipelines in Kubernetes.

This is the simple implementation of Apache airflow in a Kafka Project

A simple demonstration of an Airflow-Kafka-Spark (AKS) stack for online time series forecasting.

Benchmarks for data processing systems: Pathway, Spark, Flink, Kafka Streams

A real-time data streaming for City Residence Project

A comprehensive data engineering pipeline has been established to coordinate the ingestion, processing, and storage of data. This pipeline utilizes Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and FastDBs. All these components have been containerized with Docker to facilitate straightforward deployment and scalability

Use Pyspark To Build Kafka Streaming Application.

A Brain Computer Interface for Electroencephalogram data learning and prediction. Implementaton of Common Spatial Patterns algorithm from scratch.

Lab Works Related to the Course Big Data Analytics - S5

🎏 Python Stream Processing (Faust like!) backed by pydantic.

A scalable python microservice application built with python and flask. Ran using Docker, and utilized Kafka for real-time data streaming.

Kafka and ML use-case. Applicable to wide variety of problems for real-time recognition

A complete data pipeline, from data extraction to storage, using a combination of tools for specific purposes: Python for data retrieval from API, Airflow for scheduling task, Kafka for data streaming, Spark for data processing, and Cassandra for data storage.

Python Stream Processing