This project demonstrates a complete ETL pipeline for Formula 1 racing data using Azure Databricks, Delta Lake, and Azure Data Factory. It covers data ingestion, transformation with PySpark and Spark SQL, data governance with Unity Catalog, and visualization through Power BI. Designed to showcase real-world data engineering workflows in Azure.

Implemented an auto-clustering tool with seed and number of clusters finder. Optimizing algorithms: Silhouette, Elbow. Clustering algorithms: k-Means, Bisecting k-Means, Gaussian Mixture. Module includes micro-macro pivoting, and dashboards displaying radius, centroids, and inertia of clusters. Used: Python, Pyspark, Matplotlib, Spark MLlib.

Yelp Toronto User Pattern Analysis and Recommender System

A UDF to evaluate Spark-MLlib classification model using PySpark

A movie recommendation system built using Apache Spark’s ML library

Pequenos projetos utilizando ferramentas do ecossistema Apache Hadoop

This repo contains code for restuarant recommendation system for users based upon business rating value.

Demo clustering with LDA Spark MLlib

Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph p…

IPL Match Simulation using K-means Clustering and Collaborative Filtering.

Simple Content based and Collaborative Filtering Algorithms implementaion

Application that trains a classifier and predicts flight arrival delays based on past information. Uses the libraries pyspark.ml and pyspark.sql, performs feature engineering, cross-validation and tests various ML algorithms.

Perform sentiment analysis on Yelp dataset with Apache Spark

AEMO Aggregated price and demand data

Graduation project categorizes popular search phrases using Python and Spark and presents them on a website to inspire creators.

Using the Thunder Library for Image Processing with Spark ML Lib

A movie recommender system using user-based collaborative filtering algorithm.

Implementation to coursera machine learning course, some tensor flow code.

In this project I stream data and do crime classification using Spark. This dataset contains incidents derived from the SFPD Crime Incident Reporting system. The data ranges from 1/1/2003 to 5/13/2015. I do some data analysis of crime scenes in different areas and with respect to other parameters.