Predicting Cardiovascular Disease Risk Using Spark-Based Classification

This project explores the use of distributed deep learning with Apache Spark to classify flower images using Convolutional Neural Networks (CNNs). Image classification is a core task in computer vision, and CNNs are well-suited for learning hierarchical features directly from raw image data. By integrating Spark into this workflow, the project demonstrates how distributed computing can accelerate large-scale image preprocessing and model training across multiple virtual machines

I am using the publicly available Kaggle Dataset Flowers Recognition, which contains thousands of images across five categories: daisy, dandelion, rose, sunflower, and tulip. The project highlights the integration of Spark with deep learning libraries to handle big image data efficiently, while also serving as a stepping stone toward my final project on pneumonia detection from X-ray images.

Overview

I think that flower classification is a good starter problem for computer vision. By applying Spark-based distributed preprocessing and PyTorch CNN training, this project shows how scalable infrastructure can reduce training time and improve reproducibility. Running Spark across one master node and one worker node demonstrates the benefits of parallel computation for image-heavy tasks.

Ultimately, the goal is to show how Spark-based CNNs can be applied to real-world image classification problems, while preparing the foundation for more complex medical imaging tasks.

Key Features

Data Preprocessing

preprocess.py script

• Extracts image file paths and labels from the Flowers Recognition dataset.
• Creates a Spark DataFrame with shuffled samples.
• Splits into train/test CSVs for fine-tuning.

Model Approaches

train_cnn.py script

• Implements a simple CNN with two convolutional layers and a fully connected classifier.
• Trains on resized and normalized images (128x128) with an 80/20 train-test split.
• Evaluates accuracy on a held-out test set.
• Model saved as flower_cnn.pth.

Model Improvements

• Script 2: Added dropout (0.5) and data augmentation (flip, rotation, color jitter) to reduce overfitting. Slightly deeper classifier.
• Script 3: Added a third convolutional layer and batch normalization, increased classifier size. Trained for more epochs to improve underfitting.
• Script 4: Added a fourth convolutional layer, adaptive pooling, learning rate scheduler, and weight decay; applied controlled augmentation, resulting in the best test accuracy (77.5%).

Results

Script	CNN Layers	Key Changes / Features	Test Accuracy
1	2	Simple CNN, no dropout, no augmentation	65.36%
2	2	Added dropout (0.5), data augmentation, normalization, slightly deeper classifier	70.12%
3	3	Added batch normalization, third conv layer, larger classifier, more epochs	74.30%
4	4	Added fourth conv layer, adaptive pooling, weight decay, learning rate scheduler, controlled augmentation	77.50%

Distributed Cluster Setup

To handle the dataset’s scale, the project was deployed on two virtual machines (VMs):

• 1 Master Node and 1 Worker Node

Dataset

I found a dataset on Kaggle with 4242 images of flowers. It contains 5 classes of flowers including: daisy, dandelion, rose, sunflower, and tulip

Source: Flower Recognition

Reference: Kaggle. (2021). Flower Recognition Dataset [Dataset]. Retrieved from https://www.kaggle.com/datasets/alxmamaev/flowers-recognition?select=flowers.

How to Run Project

Prereqs

• Python 3.x
• Apache Spark installed on all VMs
• PySpark, NumPy, Pandas

Team Members

• Olivia Gette