Projects and studies regarding Data Engineering Area

Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University

Response to Jacek Laskowski's spark workshop exercises

This project analyzes data from 91wheels website (as of Nov 10, 2023) on electric scooters in India, reflecting the rising popularity of EVs. With 85 companies offering 288 models across 436 variants, it explores the evolving landscape, consumer preferences, and scooter specifications amidst the transition to electric mobility.

InsightfulRecruit: Unveiling the Job Market Landscape through Data Engineering

Discover personalized movie recommendations with this Flask and PySpark-based system. Easy setup for tailored movie suggestions.

IOT Sensors RealTime Analysis with Apache Spark, Kafka, Python and SQL.

Created a SparkML RandomForest model to predict total employee compensation. Queried data with SparkSQL, ran PySpark scripts to run EDA, pre-process data, and train model achieving with 0.98 R2 score.

From image to text - a handwriting recognition tool prototype using Image Classification - Deep Learning in DataBricks.

Stocks Data Analysis In DataBricks - Using SQL and Pyspark

[大数据课程作业] Bilibili 助手: 视频推荐 + 热门预测

practice of Spark on two datasets using PySpark

Enjoy exploring my data science projects!

data enginerring project - visualize visa numbers by country, time issued from japan

Workshop Big Data en Español

This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark clusters are set up within a Docker container on Azure.

This is the repository for all of my Spark projects, which include Spark NLP & Computer Vision projects.

End-to-end data engineer project

POC projects working on Cloud Platforms

Alumni Profile Matching is a project aimed at facilitating networking between graduate students and alumni with similar backgrounds and career goals. By leveraging machine learning techniques and data processing pipelines, the project aims to provide graduate students with personalized recommendations of alumni profiles to connect with.