Easily identify Sign Language alphabet letters (A-Z) or numbers (0-9) hand gestures with the use of your device camera!
ASL Recognizer uses machine learning based model that is trained for high accuracy detection in any environment.
Identify world’s most recognizable sign language basic gestures.
By recognizing each hand pose, you can combine whole words to understand what is being said in sign language.
Find all basic alphabet American Sign Language gestures in one place for easier identification and learning.
The project uses Combine, AVFoundation, SwiftGen and Apple Vision frameworks, is wrapped around Clean Swift VIP architecture and is fully open-source documented on GitHub.
The application uses American Sign Language (ASL) machine learning based recognition model that is organized, trained, and tested via Xcode Create ML tools.
While the developer does not provide any related data sets, it is worth mentioning that the data set for this specific ML model that application uses consists of images (that were used for training and validation of an aforementioned ML model) gathered from various sources found on the internet varying from ASL learning YouTube videos to publicly available ASL hand poses images data sets.
ASL Recognizer detects only alphabet letters A-Z and numbers from 0-9. Hand gesture detection output is shown within the main application interface. Each detected symbol (letter or number) is added to the current value with previously detected symbols.
The project is written on Xcode 13 for iOS/iPadOS devices from iOS 14.
Accuracy:
- Training - 95 %
- Validation - 86 %
Training data:
- 36 classes
- 11 897 items
The application uses Apple Vision framework to perform the hand gesture recognition with each camera output frame (CMSampleBuffer). Vision has hand pose visual detection request class that is used for hand pose and fingers recognition. Vision data parsing goes through VNImageRequestHandler.
Hand Pose ML model to predict a hand gesture uses MLMultiArray as input. What is amazing here is that multi-dimensional array (MLMultiArray) can be extracted from Vision image request handler (VNImageRequestHandler) output directly, which then is passed to model prediction methods (they were generated directly in Xcode's Create ML application before).
Finally, after model finishes with prediction, the following can be extracted: prediction confidence, predicted result (label), and many more!
The following errors/issues are handled (if occur - may provide an error message visually):
- No hand gesture found.
- Hand gesture prediction failure.
- Various camera input, output, preview layer setup errors.
- Critical thermal state condition.
Project relies on Clean VIP architecture (with own project modifications). Please find VIP architecture example scheme below:
More about this specific architecture here.
Additionally, the project uses a little bit of Combine framework to simplify some of the asynchronous actions.
User needs to grant these privacy permissions:
- Camera - Used for detecting and recognizing hand poses.
ASL Recognizer is fully offline. Thus, no data is being sent or received to and from any server. Data collection is not implemented in any way.
ASL - American Sign Language
ML - Machine Learning
If you have questions regarding this application or its support, please contact the developer.
Change Log is not available yet.
- Open project directory via Terminal:
cd project/directory
- Install pods:
pod install
- Open ASL-Recognizer.xcworkspace file.
App icon has been designed using resources from Flaticon.com.
MIT License
Copyright © 2022 Liudas Baronas.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.