SignVision

Real-time Urdu sign-language translator. Captures gestures via webcam, recognises one of 8 phrases with a CNN-LSTM trained on MediaPipe Holistic keypoints, then renders the Urdu translation as text and speech.

Final Year Project — Department of Computer Science, University of Karachi (2024). Supervisor: Dr. Farhan Ahmed. Team: Amir Aijaz, Taha Akber, Muneeb ur Rehman, Syed Abid Hassan.

Full project report: docs/Final_Year_Project_Report.docx.

What it does

Stage	Component
Capture	OpenCV webcam (cv2.VideoCapture)
Feature extraction	MediaPipe Holistic → 1662-dim keypoint vector per frame
Buffering	sliding window of 40 frames
Classification	Keras CNN-LSTM (models/sign_model_v3.h5) → softmax over 8 classes
Smoothing	PredictionSmoother (window=30, threshold 0.65/0.8, 1.5 s cooldown)
Translation	googletrans → Urdu
Speech	gTTS → MP3 saved to predicted_audio/

Supported phrases

Good morning, How are you, I am fine, Thank you, What is your name, Nice to meet you, Can you help me, Listen to me.

Results

Trained for 173 epochs with early stopping. Loss: categorical cross-entropy. Optimiser: Adam with LR scheduler.

Metric	Value
Training accuracy	95.06%
Validation accuracy	86.89%
Test accuracy	81.25%
Training loss	0.2057
Validation loss	0.5721

Full training run and evaluation cells: notebooks/dataExtract_and_model_training.ipynb.

Repo layout

SignVision/
├── src/
│   ├── realtime_inference.py   # webcam demo (entrypoint)
│   ├── translations.py         # offline English→Urdu table (8 phrases)
│   └── tflite_converter.py     # H5 → TFLite for mobile handoff
├── scripts/
│   └── collect_data.py         # capture new keypoint sequences from webcam
├── tests/
│   └── test_keypoints.py       # dataset + translation table smoke tests
├── models/
│   ├── sign_model_v3.h5        # production model (7.2 MB)
│   └── sign_model.tflite       # fp16-quantised (1.2 MB)
├── dataset/
│   ├── README.md
│   └── MP_Data/                # 8 classes × 40 seqs × 40 frames (.npy)
├── notebooks/
│   └── dataExtract_and_model_training.ipynb
├── docs/
│   └── Final_Year_Project_Report.docx
├── requirements.txt
├── LICENSE
└── README.md

Running it

Python 3.10 required (TensorFlow 2.10 + MediaPipe 0.10).

git clone https://github.com/amiraijaz/SignVision.git
cd SignVision
python -m venv .venv && .venv\Scripts\activate   # Windows
pip install -r requirements.txt

python src/realtime_inference.py
# or:
python src/realtime_inference.py --model models/sign_model_v3.h5 --camera 0

Press q to stop. The combined Urdu audio is written to predicted_audio/combined_predictions_<timestamp>.mp3.

Translation uses an offline Urdu lookup table for the 8 known phrases (src/translations.py), so the demo doesn't depend on internet for translation. gTTS still needs internet to synthesise the MP3.

Tested on Windows 10, Python 3.10, TensorFlow 2.10.

Tests

pytest tests/

Checks that the dataset has all 8 classes, that keypoint vectors are 1662-dim, sequences are 40 frames, and the translation table covers every action.

Collecting new data

python scripts/collect_data.py "New phrase" --sequences 40 --frames 40

Writes to dataset/MP_Data/New phrase/<0..39>/<0..39>.npy.

Re-training

Open notebooks/dataExtract_and_model_training.ipynb. The notebook covers:

1. Live data collection into dataset/MP_Data/
2. Sequence loading and label encoding
3. CNN-LSTM model definition
4. Training, evaluation, confusion matrix
5. Saving to models/sign_model_v3.h5

To regenerate the mobile TFLite model:

python src/tflite_converter.py

Scope of this repository

This repo covers the AI / data contribution: dataset collection, model training, and the real-time inference backend. The mobile application, authentication backend, and Arduino hardware prototype described in the report are tracked in separate deliverables and are not part of this repo.