Deaf children are often born to hearing parents who do not know sign language. The challenge in this code competition is to help identify signs made in processed videos, which will support the development of mobile apps to help teach parents sign language so they can communicate with their Deaf children.
The competition aims to develop a TensorFlow Lite model for classifying isolated American Sign Language (ASL) signs. The model will be trained on labeled landmark data extracted using the MediaPipe Holistic Solution. The goal is to improve the functionality of the PopSign smartphone game app, which helps parents of deaf children learn ASL and allows anyone interested in learning sign language vocabulary to practice.
By creating a sign language recognizer, participants can enhance the game's interactivity and facilitate learning and communication for players who want to use sign language to connect with their loved ones. Since the app doesn't send user videos to the cloud and performs all inference on the device itself, TensorFlow Lite is the chosen framework to ensure low latency within the game.
google_asl
├── utility_scripts/
| ├── feature_engineering.py
| └── helper_functions.py
├── visualization_scripts/
| ├── animated_3d_viz.ipynb
| └── viz_dashboard.ipynb
├── datasets/
| ├── mean_dataset.npy
| ├── feature_data.npy
| └── feature_labels.npy
| ├── spatio_temporal_data/
| ├── spatio_temporal_script.ipynb
| ├── feature_data.npy
| └── feature_labels.npy
| └── kaggle_data/
| ├── sign_to_prediction_index_map.json
| ├── train.csv
| └── train_landmark_files
├── preprocessing_keras_layer/
| └── angular_distance_layers.py
├── models/
| ├── gru_asl_model.ipynb
| ├── nn_asl_model.ipynb
| └── sequential_model.ipynb
└── application/
| ├── mediapipe_landmarks.py
| └── real_time_asl_recognition.ipynb The goal of this first step is to create a new dataset that takes into consideration both the temporal and spatial aspect of the ASL training set to optimize the performance of our model.
This part presents an interactive dashboard that allows exploring American Sign Language (ASL) landmarks in 2D space. By selecting different ASL categories and IDs, users can filter the displayed landmarks and study the hand shapes and movements associated with specific signs. The dashboard is built with Plotly and Python, and can be used for educational purposes or to support the development of ASL recognition systems.
Dashboard.Vizualisation.mov
The goal here is to build a baseline ASL recognition engine using the average frames dataset. We will be leveraging the modularity of the custom keras layer to apply some preprocessing steps to the 3D landarks and for feature extraction.
Use your pre-trained TensorFlow Lite model to perform real-time American Sign Language (ASL) recognition from YOUR webcam video!
