Probably my first project where I was also responsible for data collection :)
Skills and tools - Computer Vision, Gaussian Mixture model, Mediapipe, PCA, Machine Learning, Python
Spelt my name using ASL sign language alphabets :)
Used mediapipe to get hand landmarks from running webcam videostream .
Substracted first pair of coordinates from all others, so that my values only depend on the relative position of fingers rather than their position across the screen .
(Another idea that can be implemented in future is maybe getting the angles between the fingers , we know that angle between 2 lines is |m1-m2|/|1+m1m2| so we can use that )
I have only trained for 5 alphabets - A, Y, U, S, H (the alphabets in my name) since data collection would have been a long process and this was meant to be a fun project .
Here is a GIF of me collecting data
Data Collection code - https://github.com/ayush-agarwal-0502/Sign-lang-translation-using-GMM/blob/main/data_collector_sign_language_translator_1.ipynb
After this process I got a dataset that looked like this -
These are the PCA plot results of the dataset . Makes sense to me .
Would love it if someone wants to collaborate on the data colection part and wants to collect the data for all the alphabets . Maybe we can make something nice for the deaf ppl.
My hypothesis was that the 42 datapoints for hand would have different values for each alphabet , the values would be somewhat gaussian around the mean (where the mean represents the perfect way to hold up that alphabet in sign language) . PCA showed that my hypothesis was somewhat right . So I proceeded with training a GMM (Gaussian Mixture Model) .
Data Processing Code - https://github.com/ayush-agarwal-0502/Sign-lang-translation-using-GMM/blob/main/model_sign_language_translator_2.ipynb
For deployment I decided that if the model was 99% or more sure about an alphabet (bcoz its a gaussian so we can easily calculate confidence wrt a prediction) then it would show an alphabet else it would show nothing .
Also here are some other mediapipe tools I was playing with for fun, u might find it intresting -
https://github.com/google-ai-edge/mediapipe/blob/master/docs/solutions/face_detection.md
https://github.com/google-ai-edge/mediapipe/blob/master/docs/solutions/holistic.md
