Hand Signal Computer Vision
Table of Contents
HandSignalCV is a project designed to detect American Sign Language (ASL) from a camera feed and convert it into text. The project will utilize OpenCV and Mediapipe to capture frames and retrieve an ASL dataset. This dataset will then be divided into training, validation, and test sets.
Using this data, the project will create a Deep Neural Network (DNN) with convolutional layers and ReLU activation functions, trained using the Adam optimizer. An AI API will be integrated to verify the constructed sentences, ensuring accuracy.
Python, Jupyter Notebook Libraries: OpenCV, Mediapipe, numpy, tensorflow, sk-learn, h5py
pip install tables pip install h5py pillow pip install jupyter pip install tensorflow python -m pip install cohere --upgrade pip install python-dotenv
- Clone the repo
git clone https://github.com/wirefailed/HandSignalCV.git
-
Run capturingSignals.py
python capturingSignals.py
During each iteration of the run, write the alphabets and press 's' to save the photo. Hold it, and it will automatically close when it has enough data or if the program crashes. This step is crucial for later application because, even if the CNN model classifies it properly, it will not be effective if the data is incorrect in the first place.
-
Run python hdf5_conversion.py
python hdf5_conversion.py
Simply run the code and it will automatically create hdf5_file for you.
-
Get Cohere API Key and create .env file
echo "CO_API_KEY=your_actual_api_key_here" > .env
This will echo CO_API_KEY=your_actual_api_key_here in .env. It will automatically create .env file if it does not exist. Simply change 'your_actual_api_key_here' to Cohere API Key
-
Run detectingSignal.py
python detectingSignal.py
Run the following code and it will start displaying the letter on the monitor. Press 'c' and it will start saving the letter by given interval (changeable). Press 's' to stop and print the statement on terminal.
For individuals who don't understand American Sign Language (ASL), this tool allows ASL users to communicate by using hand signals in front of a computer, which then converts these signals into written sentences. This makes it easier for non-ASL users to read and understand hand signals conveniently.
- Install OpenCV and mediapipe
- Using both libraries, set it to detect hand and display handlandmarks
- Create a rectangle around the hand using cv2.rectangle
- Crop the rectangle seperately
- Use its size to get lmList and get resized_lmList that fits into white square 300 by 300
- Capture hand signals from a to z 200 each and split data samples 6:2:2 ratio
- Train the model and optimize it to have >85% accuracy
- Deploy the system and connect into AGI api (if not possible, skip this step)
Distributed under the MIT License. See LICENSE.txt for more information.
Junsoo Kim
Project Link: https://github.com/wirefailed/HandSignalCV.git