✋ Indian Sign Language Recognition to Speech
A real-time sign language recognition system that converts hand gestures into text and speech. This project uses computer vision and machine learning to help bridge the communication gap between sign language users and those who don’t understand sign language.
🌟 Overview
Imagine having a translator that can watch your hand movements and instantly convert them into spoken words. That’s exactly what this project does:
📷 Captures video from your computer’s camera.
🤖 Recognizes sign language letters (currently supports A, C, and T).
📝 Displays recognized letters as text in real time.
🔊 Converts text to speech using AI voice.
⚡ Runs live and instantly (low latency, ~30 FPS).
This project demonstrates how AI-powered assistive technology can support deaf and hard-of-hearing communities, while also being an educational tool for learning computer vision and sign language.
💡 Why This Project Matters
Sign language is used by millions worldwide, but very few non-signers understand it. In India especially, tools for Indian Sign Language (ISL) are scarce.
This project:
Promotes inclusion for deaf and hard-of-hearing individuals.
Makes education and workplaces more accessible.
Helps non-signers communicate with ISL users.
Provides a hands-on learning platform for AI, HCI, and accessibility research.
🚀 How It Works
Think of this as teaching a computer to "see" and "understand" hand gestures:
-
👁️ Seeing → Camera captures hand movements.
-
🔍 Understanding → MediaPipe detects 21 hand landmarks (fingertips, joints, palm).
-
🧠 Learning → AI model (Random Forest or SVM) recognizes gestures.
-
🗣️ Speaking → Converts recognized gestures into text and speaks aloud.
📂 Project Structure
sign-language-recognition/ │── 1.data.collection.py # Collects training images for gestures │── 2.HandLandmarks.py # Extracts hand landmarks from images │── 3.TrainModel.py # Trains AI models (Random Forest / SVM) │── 4.Deployment.py # Main real-time translator │── 4.1Deployment-2-frame.py # Enhanced version with better detection │── data/ # Folder for gesture image datasets │── model.p # Trained ML model (generated after training) │── data.pickle # Processed landmark data (generated)
🔧 Step-by-Step Setup
- Install Python
Download and install Python 3.8+ from python.org. (Ensure you check “Add Python to PATH” during installation.)
- Install Dependencies
Run the following in your terminal:
pip install opencv-python mediapipe scikit-learn numpy matplotlib tkinter gtts pickle
- Collect Training Data
Capture your own gesture images:
python 1.data.collection.py
Show gestures for A, C, and T.
System captures 100+ images per gesture.
- Process Data (Extract Hand Landmarks)
python 2.HandLandmarks.py
Converts images into numerical hand landmark data.
Creates data.pickle.
- Train the AI Model
python 3.TrainModel.py
Trains both Random Forest and SVM classifiers.
Saves best model as model.p.
- Run the Translator
python 4.Deployment.py
Opens camera feed.
Recognizes signs → Converts to text → Speaks aloud.
For better accuracy, run the enhanced version:
python 4.1Deployment-2-frame.py
🎮 How to Use
Show gesture like A, C, or T to the camera.
Recognized letter will appear in text box.
Use GUI buttons:
🟧 Clear Text – Reset text box.
🟩 Speak Text – Speak letters aloud in English.
🟪 Translate Speech – Speak in French.
🟥 Exit – Close application.
📊 Performance Metrics
✅ Accuracy: 85–95% (depends on dataset quality).
⚡ Processing speed: ~30 FPS.
⏱️ Response time: < 1 second.
💾 Memory usage: ~200 MB during runtime.
🕐 Training time: 2–5 minutes for 3 gestures.
🌍 Applications
This system can be used in:
🏫 Education → Teach/learn sign language.
🏥 Healthcare → Doctor-patient communication.
🛒 Customer Service → Accessible counters/kiosks.
🚉 Transportation → Public info kiosks.
💻 Video Conferencing → Real-time ISL translation.
🎮 Gaming & HCI → Gesture-controlled interfaces.
🔮 Future Enhancements
Full A–Z alphabet support.
Recognition of words and phrases.
Support for Indian Sign Language (ISL) grammar.
Mobile app versions (Android/iOS).
Cloud training & sharing of datasets.
Real-time conversational translation.
📖 Educational Value
This project demonstrates:
Computer Vision → Hand landmark detection with MediaPipe.
Machine Learning → Training classifiers for gesture recognition.
Real-Time AI → Running inference on live video.
Accessibility Technology → Practical assistive tools for communication.
Python Development → End-to-end ML + HCI project pipeline.
🤝 Contributing
You can improve this project by:
Adding more gestures (letters/words).
Improving the UI/UX.
Enhancing accuracy with deep learning (CNNs/LSTMs).
Creating mobile/web app versions.
Supporting multi-language speech synthesis.
🔐 Privacy & Security
All recognition happens locally on your computer.
Works offline (except text-to-speech).
No images are stored permanently unless you save them.
🎓 Learning Opportunities
By working with this project, you’ll gain hands-on experience with:
Python programming.
Data collection and preprocessing.
ML model training and evaluation.
Real-time video processing.
GUI development for accessibility.