American Sign-language to Audio 😎

This repo hosts the code for a system that can convert American sign language to audio. As the title suggests, this system aims to establish efficient communication between normal language and sign language.

Technologies used 💖

• OpenCV
• TensorFlow
• gTTS

About the system 🎶

• The core of the system is a machine learning model (find the notebook 🌟 here 🌟 ).
• An example dataset for the ML model can be found here. The images undergoes the following transformations before being used for training:
  👇👇👇

  gray = cv.cvtColor(image, cv.COLOR_BGR2GRAY)
  blur = cv.GaussianBlur(gray,(5,5),2)
  th3 = cv.adaptiveThreshold(blur, 255, cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY_INV, 11, 2)
  ret, res = cv.threshold(th3, 70, 255, cv.THRESH_BINARY_INV + cv.THRESH_OTSU)
  res = cv.resize(res, (128,128))

• After training the model to the desired accuracy, you can save it in Custom_models folder. Save it under the name: base_model.h5 👏 👏
• After the initial training procedure, the model can further be trained by transfer learning on the base model on the custom dataset of a person using detect_and_train.py. You can run the python file using the command:

root@Gokuls-Laptop:~/Sign-language# python self_train/detect_and_train.py -n [NAME]

where [NAME] is the name of the user whose database was just created. This python file will capture 100 images of each of the letters done the the person's hands 💦 💦 . This database will be then used for transfer learning and the model will be saved with the persons name in Custom_models.

• After training for the profile, run the python file hand_track.py using the following command:

root@Gokuls-Laptop:~/Sign-language# python hand_track.py -n [NAME]

where [NAME] is the name of the person whose database already exists. To use the base model itself, the arguments can be skipped 💔 and simply run:

root@Gokuls-Laptop:~/Sign-language# python hand_track.py

Find the notebook for the base model here. 💞

Model Details

• Framework: TensorFlow
• Architecture: Convolutional Neural Network.
• Recommends training till an accuracy of 80% on the training data atleast (can be improved with detect_and_train.py)

Gokul G Menon