SignSpeak is a cross-platform American Sign Language (ASL) recognition platform that:
- 🕒 Detects ASL signs in real-time via your device camera
- 📱 Runs on mobile devices (iOS & Android) using React Native + Expo
- 🤖 Processes frames on a Python Flask backend with MediaPipe, OpenCV, and trained ML models
- 🏗️ Dockerized for one‑command deployment (development or production)
🎥 Model Demo: Watch proof video
-
Real-Time Recognition: 30+ frames per second hand detection with MediaPipe
-
Multiple Strategies:
- Keypoint-based classification
- CNN-powered image classification
- Optional YOLOv5 object detection pipeline
-
Session Recording: Start/stop sessions, save raw frames, export metadata
-
Cross-Platform UI: Shared codebase for iOS, Android, and Web via Expo
-
Scalable Backend: Containerized Flask API, ready for production with Gunicorn + Nginx
SignSpeak/
├── UI_expo/ # React Native (Expo) mobile app
│ ├── components/ # UI & camera overlay
│ ├── services/ # API wrappers & data handling
│ └── app.json # Expo config
├── server/ # Flask backend API
│ ├── App.py # Main application
│ ├── recordings/ # Stored session data
│ └── requirements.txt
├── models/ # Pretrained ML models & label files
│ ├── random_forest_model.pkl
│ └── label_classes.txt
├── deploy/ # Docker & Nginx configs
│ └── docker-compose.yml
├── assets/
│ ├── SignSpeak.png # README banner
│ ├── ASL Alphabet.jpg # ASL chart reference
│ └── Model_proof.mp4 # Demo video
├── docs/ # (Optional) Extended documentation
├── Scripts/ # Training & preprocessing scripts
└── README.md # This file
-
Clone the repository
git clone https://github.com/CodeWithInferno/SignSpeak.git cd SignSpeak -
Backend
cd server python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python App.py
The API will run at:
http://localhost:5050 -
Mobile App
cd UI_expo npm install npx expo start- Open the Expo Go app on your phone and scan the QR code, or launch on an emulator.
-
Start with Docker Compose
docker-compose up --build -d
-
Verify services
docker-compose ps
-
Access UI on your LAN at
http://<your-host-ip>:19006
-
Banner:
./SignSpeak.png -
ASL Alphabet Chart:
-
Model Proof Video:
| Layer | Technology |
|---|---|
| Frontend | React Native (Expo) |
| Backend | Python, Flask, Flask-CORS |
| CV/ML | MediaPipe Hands, OpenCV, scikit-learn (RF), YOLOv5 |
| Deployment | Docker, Docker Compose, Gunicorn |
| Platform | Android, iOS & Web (Progressive) |
Contributions are welcome! Please:
- Fork the repo
- Create a feature branch (
git checkout -b feature/YourFeature) - Commit your changes (
git commit -m "Add YourFeature") - Push (
git push origin feature/YourFeature) - Open a Pull Request
For major changes, open an issue first to discuss what you’d like to change.
This project is licensed under the MIT License. See LICENSE for details.
- GitHub Issues: codewithinferno/signspeak
- Discussions: GitHub Discussions
- Email: praathambiren2618@gmail.com
Made with ❤️ by Pratham Patel, Raahil Desai & Tashvi Patel