scripts for starting up a kafka+spark cluster on a slurm manager

Official repo for DK908 - IOT Big Data Processing

A Data pipeline and a analytics dashboard that tracks the number of data updates in interval of 1 mins. The result is shown as a graph on the web browser. The input data is first supplied to Kafka, from which it gets streamlined to Spark streaming where some processing happens. Then the final data is updated on the browser.

A repository for ipython notebook backup

Big Data Analytics for smart shopping cart using Spark Framework and Scala Programming Language

Structured Spark Streaming workshop for Codess 2018

This project will show an auto-updated map with the people interaction during COVID19 in the US using big data technologies to analysis a real-time stream of Twitter data.

End-to-end real-time credit card transactions application. Made with Kafka, Spark, Bootstrap, ECharts, RxJS.

Analysis for a streaming daily retail data using Spark structured streaming and querying this data to get insights

My Technical Blog Repo

Just a few proyects with py, r, spark, storm, and others...

MBIT Big Data 2019-2020 Apache Spark Case Study (M-02 DC-02)

This project uses machine learning to categorize and prioritize airline user tweets based on content and sentiment. The goal is to reduce airlines' workload and provide personalized, empathetic responses to users. By training a sentiment analysis model, airlines can better understand customers' needs and improve their overall service on Twitter.

As a Data Engineer for a fictional E-commerce startup, this project addresses the task of analyzing the web server logs to find the number of product pages visited and the number of items in the cart.

Python projects related to the course- Algorithms for Data Guided Business Intelligence

Jupyter notebooks for filtering Kafka data with Spark Streaming.

Statistics of coin’s volume in real time on Binance in the last 1 hour. Use Kafka to get data from Binance API, then Spark Streaming reads data stream.

基于Spark的电影推荐系统

Twitter Spark Streaming using PySpark