Tools Used:
- Spark SQL
- Spark Machine Learning
- H2O ML, Deep Learning and AutoML
- AWS EMR Clusters
- Plotly
Data Source: https://www.kaggle.com/benhamner/sf-bay-area-bike-share Data Size: 4GB
- station.csv - Contains data that represents a station where users can pick up or return bikes.
- status.csv - Data about the number of bikes and docks available for a given station and minute.
- trips.csv - Data about individual bike trips
- weather.csv - Data about the weather on a specific day for certain zip codes
Predict number of bikes available at a given station with:
- station information
- weather condition
- type of day (weekday/weekend)
- hour of the day
- population of the neighborhood
Data Pipeline:
- Exploratory Data Analysis
- Data Visulization with Plotly
- Data Preprocessing + Spark ML Models on AWS EMR Clusters
- H2O ML, Deep Learning, and Auto ML
Prediction Result:
- For Spark ML Models, our best model is Random Forest, with an RMSE of 2.7
- For H2O Models, AutoML's XGBoost model has an RMSE of 2.38
Run Time Comparison on different AWS EMR Clusters:
Group member: Esther Liu, Marine Lin, Akankasha, Lexie