Sign Language Recognition iOS App

An iOS application for real-time sign language recognition using on-device machine learning with Core ML.

Features

• Real-time sign language detection and translation
• Uses Vision and Core ML frameworks for efficient on-device inference
• Support for dark and light mode with adaptive UI
• Detailed recognition results with confidence scores
• Motion detection for improved gesture recognition
• Supports 500+ sign language gestures

Technical Details

This application uses:

• Swift and SwiftUI for the user interface
• Vision framework for pose detection
• Core ML for on-device inference
• ONNX model format for the sign language recognition model
• Adaptive UI that works on different iPhone screen sizes

Getting Started

Prerequisites

• Xcode 13.0 or later
• iOS 15.0 or later
• An ONNX sign language recognition model
• A JSON file with label mappings

Setup

1. Clone the repository
2. Open the project in Xcode
3. Run the app on your iOS device
4. On first launch, you'll need to import the ONNX model and labels file
5. Once setup is complete, you can start using the sign language recognition

How It Works

The app processes video frames from the camera to detect body and hand poses using Apple's Vision framework. These poses are then mapped to a format compatible with the sign language recognition model. The model analyzes sequences of poses to recognize sign language gestures, providing real-time translations.

Performance

• Inference time: ~1.2 seconds per gesture
• Low memory footprint
• Battery-efficient processing

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• ONNX Runtime for efficient model execution
• Apple's Vision framework for pose detection
• The sign language dataset contributors