Leverages Apache Kafka to facilitate streaming real time data generated by Python to upload data into S3 using s3fs

My Talk at IoT Fusion 2018 Philadelphia, PA

Playing with Apache Kafka and following tutorials to get a feel for it.

Udacity Data Streaming project based on Apache Kafka

Big Data Technologies can be defined as software tools for analyzing, processing, and extracting data from an extremely complex and large data set with which traditional management tools can never deal

This project focuses on building a real-time streaming pipeline using Apache Flink and Apache Kafka. The goal is to enrich checkout data with user information, identify the first click leading to a checkout, and log the attributed checkouts into a Postgres sink table. The project implements concepts like state management, time attributes, watermark

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage. All components are containerized with Docker for easy deployment and scalability.

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

An app to keep track of Youtube videos and sends the notification to a Telegram bot to inform you if anyone comments on those

O Crawler Shein é um projeto de Web Scraping automatizado desenvolvido usando Scrapy para extrair informações detalhadas de produtos no site Shein .

The code repository encompasses a real-time election voting system constructed with Python, Kafka, Spark Streaming, Postgres, and Streamlit. Docker Compose is employed to effortlessly launch the necessary services within Docker containers.

Simple local running Apache Kafka implementation using confluent-kafka

Udacity Data Streaming Nanodegree Projects

Repositório destinado a estudos referente apache-kafka

Worked Example Miner (WEM): A Comprehensive Tool for Analyzing Java Repositories.

Project Ember official city simulator (scale to city levels your test environment)

Wordcloud using Twitter API and Apache Kafka

Simple implementation of a message streaming social network using Apache Kafka

The goal of the project is to deal with the real-time streaming analysis and study how it works by implementing data pipelines.