Signer is an Attention-Based Multi-Fusion deep learning model that translates sign language videos into text. It combines computer vision (MediaPipe landmark extraction) with a multi-modal deep learning architecture and Large Language Models to process videos, extract meaningful patterns from gestures, and produce text translations.
Pipeline Overview:
Input Video → Boundary Detection → Landmark Estimation → Feature Extraction → Base Model → Gloss Predictions → LLM + Prompt Engineering → Output Text
-
🎥 Gloss Recognition — Predicts the full sequence of sign actions (glosses) from video using boundary detection to split continuous signing into individual signs.
-
🖐️ Feature Extraction — Uses MediaPipe to detect and track hand, pose, and facial landmarks. Extracts hand images, spatial region codes, hand-face distances, and landmark coordinates from each frame.
-
⚡ Cross-Attention Fusion — Fuses image features (MobileNetV2 backbone) with pose/landmark features through bidirectional cross-attention, where each modality attends to the other.
-
🤖 Transformer Encoder — Models temporal dependencies across video frames using a Transformer Encoder with learned positional embeddings.
-
🌍 Translation to Text — Converts recognized gloss sequences into coherent text via LLM-based prompt engineering.
| Pipeline | Description |
|---|---|
| InferencePipeline | End-to-end gloss recognition: takes a video path, detects sign boundaries, extracts features, and returns top-5 predicted glosses with probabilities. |
| TrainingPipeline | Loads pre-processed video tensors, extracts mediapipe features, and trains the model with augmentation support. |
| PreprocessingPipeline | Converts raw video files into frame tensors organized by gloss folders, with optional compression. |
| MediapipeVisualizationPipeline | Visual debugging tool that demonstrates each processing step and allows direct feature extraction. |
# Clone the repository
git clone https://github.com/Almahil249/Signer.git
cd Signer
# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies (recommended: Python 3.12.11)
pip install -r requirements.txtSigner/
├── InferencePipeline/
│ ├── BoundaryModel.py # Transformer-based sign boundary detector
│ ├── Classifier.py # Multi-partition majority voting classifier
│ ├── Inference.py # Main inference pipeline orchestrator
│ ├── Model.py # Multi-modal fusion model (CrossAttention + Transformer)
│ ├── VUManager.py # Video upload manager (frame extraction + prediction)
│ └── featureExtractor.py # MediaPipe landmark extraction + feature processing
├── MediapipeVisualizationPipeline/
│ ├── Extractor.py # Landmark extractor with visualization support
│ ├── FeaturesExtractorFromImage.py # Per-frame spatial feature extraction
│ ├── MediapipeFeatureExtractor.py # Video-level feature extraction with fill-forward
│ ├── VideoCropper.py # Person detection and frame cropping
│ ├── VideoProcessor.py # Video processing orchestrator
│ └── Visualizer.py # Visualization pipeline entry point
├── PreprocessingPipeline/
│ ├── Compressor.py # Batch compression of gloss folders to tar.gz
│ └── Preprocess.py # Video-to-tensor conversion and gloss folder creation
├── TrainingPipeline/
│ ├── MediapipeFeaturesExtractor.py # Training-time feature extraction (ImageCropper + features)
│ ├── Model.py # Training model (same architecture as inference)
│ ├── VideoDataset.py # PyTorch Dataset with augmentation support
│ ├── tools.py # Training loop, prediction, and evaluation utilities
│ └── example.ipynb # Training example notebook
├── Data sample/
│ ├── sample.jpeg # Sample image for landmark checking
│ └── sample.mp4 # Sample video for inference testing
├── source/
│ ├── Inference Notebook.ipynb # Original inference notebook
│ └── Training Notebook.ipynb # Original training notebook
├── inferencePipelineExample.py # Inference pipeline usage example
├── visualizationPipelineExample.py # Visualization pipeline usage example
├── preprocessor.py # Preprocessing pipeline usage example
├── requirements.txt # Python dependencies
└── .gitignore
See the example files for each pipeline:
You can also run the notebooks directly on Kaggle without local setup — Kaggle handles loading the data and saved model weights:
We used the AUTSL Dataset (Ankara University Turkish Sign Language Dataset), a large-scale multimodal dataset containing 226 isolated Turkish signs performed by 43 signers across 38,336 video samples. Samples are recorded using Microsoft Kinect v2 at 512×512 resolution. Access the dataset from the AUTSL website.
For this project, we used only the RGB format and selected 26 glosses. The model achieved a Top-1 accuracy of 91% on these glosses. Training took approximately 2 weeks on Kaggle.
Selected Glosses (26)
| Index | Gloss | Index | Gloss |
|---|---|---|---|
| 0 | accident | 13 | goodbye |
| 1 | always | 14 | hurry |
| 2 | apologize | 15 | police |
| 3 | bed | 16 | same |
| 4 | belt | 17 | sibling |
| 5 | breakfast | 18 | single |
| 6 | bring | 19 | thanks |
| 7 | forbidden | 20 | time |
| 8 | friend | 21 | tomorrow |
| 9 | full | 22 | wait |
| 10 | get_well | 23 | where |
| 11 | glove | 24 | who |
| 12 | good | 25 | why |
Pre-trained weights are available on Kaggle:
We welcome contributions! Whether you want to fix bugs, improve documentation, add new datasets, or extend the translation models — please fork the repo, create a feature branch, and submit a pull request.
This project is licensed under the MIT License.
- Magdi Waleed – GitHub | LinkedIn | Email
- Ahmed S. Mahil – GitHub | LinkedIn | Email
- Ahmed Mohammed – GitHub | LinkedIn | Email
Feel free to contact us for more details.