git clone https://github.com/wnukmat/ASL_Gesture_Recognition
https://www.kaggle.com/datamunge/sign-language-mnist/data <br/>
https://www.bu.edu/av/asllrp/dai-asllvd.html <br/>
- Mansur Amin - CNN Images - mansuramin
- Juan Castillo - Transfer Learning Videos - juancastillo
- Matthew Wnuk - LSTM Videos - wnukmat
In this project we propose using deep learning to implement a system that can identify/classify both static
images and gestures of mono-morphemic signs from videos. We will be using 2 strategies that leverage the advances
in deep learning .The first strategy employs the use of a single Convolutional Neural Network that will classify
letters of the alphabet and will be known as the ASL Image Model. The second strategy employs the use of transfer
learning, where we attempt at using two network architectures to classify ASL videos, this model will be known as
ASL Video Models. The first being a CNN paired with a attention mapping architecture, and the second being a CNN
paired with an Recurrent Neural Network (RNN).
Install packages as follows:
pip install os
pip install cv2
pip install time
pip install numpy
pip install pandas
pip install matplotlib.pyplot
The main.py file will create a set of directories filled with the training data and train all the associated models for this project. Runtime is approximately 3.5 hours and will produce plots found in the report. To run code:
python main.py
main.py -- Extracts Data and Trains all Models
asl_img_cnn/cnn_asl_mnist.ipynb -- Trains and Test using Kaggel Image Dataset
asl_img_cnn/cnn_asl_mnist.py -- .py version of cnn_asl_mnist.ipynb
asl_video_lstm/Create_Image_Directories.ipynb -- Creates images directories to be used for single frame classifier
asl_video_lstm/clean_data.py -- Crops the videos and produces two new videos
asl_video_lstm/getFeatures.ipynb -- Creates and saves features for LSTM model
asl_video_lstm/get_data.py -- Download data from Online
asl_video_lstm/load_features.py -- Loads features from directories to be used for LSTM training
asl_video_lstm/lstm.py -- Trains LSTM
asl_video_lstm/test.xlsx -- Excel file pointing to online videos and used by get_data.py