🤟 Sign Sarthi - Indian Sign Language Detection Platform

┌─────────────────────────────────────────────────────────────────┐
│                         User Browser                             │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────────┐        │
│  │  Webcam     │  │  MediaPipe   │  │  Web Speech API │        │
│  │  Feed       │→ │  Hand Track  │→ │  (TTS/STT)      │        │
│  └─────────────┘  └──────────────┘  └─────────────────┘        │
│         ↓                ↓                    ↓                  │
│  ┌──────────────────────────────────────────────────────┐       │
│  │           JavaScript Frontend Controller              │       │
│  │  • Frame Processing  • Gesture Detection             │       │
│  │  • Optimistic Updates • Real-time Display            │       │
│  └──────────────────────────────────────────────────────┘       │
└────────────────────────────┬────────────────────────────────────┘
                             │ HTTPS/WebSocket
                             ↓
┌─────────────────────────────────────────────────────────────────┐
│                    Flask Backend Server                          │
│  ┌───────────────────────────────────────────────────────┐      │
│  │              RESTful API Endpoints                     │      │
│  │  /process_frame  /add_character  /clear_sentence      │      │
│  └───────────────────────────────────────────────────────┘      │
│         ↓                    ↓                  ↓                │
│  ┌─────────────┐   ┌─────────────────┐   ┌──────────────┐      │
│  │  MediaPipe  │   │  TensorFlow/     │   │  Translation │      │
│  │  Processor  │→  │  Keras Model     │→  │  Service     │      │
│  └─────────────┘   └─────────────────┘   └──────────────┘      │
│                            ↓                                     │
│  ┌──────────────────────────────────────────────────────┐       │
│  │          Global State Management                      │       │
│  │  • Predicted Text Storage                            │       │
│  │  • Session Handling                                  │       │
│  └──────────────────────────────────────────────────────┘       │
└─────────────────────────────────────────────────────────────────┘
                             ↓
┌─────────────────────────────────────────────────────────────────┐
│                   Docker Container (Hugging Face Spaces)         │
│  • Python 3.10 Runtime                                           │
│  • Gunicorn WSGI Server (2 workers, 4 threads)                  │
│  • Port 7860 Exposed                                             │
└─────────────────────────────────────────────────────────────────┘

Webcam → Canvas Capture → Base64 Encoding → POST /process_frame 
  → MediaPipe Hand Detection → Landmark Extraction → Feature Engineering 
  → Model Prediction → Character Return → Frontend Display Update

Text Input → Character Split → Video Mapping → Sequential Playback 
  → Animation Display → Completion Callback

Recognized Text → Google Translate API → Target Language 
  → Formatted Output → Display

ISL/
├── 📄 app.py                      # Main Flask application (413 lines)
│   ├── Route handlers (/process_frame, /add_character, etc.)
│   ├── MediaPipe initialization
│   ├── Model loading and prediction logic
│   └── Global state management
│
├── 🤖 model.h5                    # Trained Keras model (~11.5 MB)
│   └── Sequential CNN for gesture classification
│
├── 📋 requirements.txt            # Python dependencies
│   ├── Flask==3.0.0
│   ├── tensorflow==2.15.0
│   ├── mediapipe==0.10.9
│   └── opencv-python==4.8.1.78
│
├── 🐳 Dockerfile                  # Docker container configuration
│   ├── Base: python:3.10-slim
│   ├── System dependencies (libgl1, ffmpeg, etc.)
│   ├── Python packages installation
│   └── Gunicorn server setup
│
├── 🌐 templates/                  # HTML templates
│   ├── index.html                # Landing page with features overview
│   ├── ISL.html                  # Real-time ISL detection interface (1052 lines)
│   ├── text_to_isl.html         # Text-to-ISL converter
│   ├── about.html               # About page
│   └── community.html           # Community resources
│
├── 🎨 static/                     # Static assets
│   ├── assets/
│   │   ├── css/                 # Stylesheets
│   │   ├── js/                  # JavaScript libraries
│   │   ├── images/              # UI images and icons
│   │   └── webfonts/            # Font files
│   ├── images/                  # Feature images
│   ├── videos/                  # ISL alphabet video library (A-Z)
│   │   ├── a.mp4
│   │   ├── b.mp4
│   │   └── ... (26 videos)
│   └── script/
│       └── script.js            # Global JavaScript utilities
│
├── 🧠 Model/                      # Model development directory
│   └── Model/
│       ├── dataset_keypoint_generation.py  # Feature extraction script
│       ├── ISL.ipynb                      # Training notebook
│       ├── keypoint.csv                   # Training dataset
│       ├── main.py                        # Model training script
│       └── model.h5                       # Backup model file
│
├── 🎬 NFSL/                       # Non-Formal Sign Language (Action Recognition)
│   ├── NFSL.py                   # Action detection script
│   ├── Action Detection Refined.ipynb  # Development notebook
│   ├── action.h5                 # LSTM model for action sequences
│   ├── MP_Data/                  # MediaPipe landmark data
│   │   ├── hello/               # Sequence data for "hello"
│   │   ├── iloveyou/            # Sequence data for "I love you"
│   │   ├── namaste/             # Sequence data for "namaste"
│   │   └── thanks/              # Sequence data for "thanks"
│   ├── Logs/                     # TensorBoard training logs
│   └── templates/
│       └── index.html           # NFSL interface
│
├── 📷 image/                      # Application screenshots
│
├── 📝 README.md                   # This file
├── 📝 README_HF.md               # Hugging Face Spaces README
├── 🔐 .gitignore                 # Git ignore rules
├── 📦 .dockerignore              # Docker ignore rules
└── 🗂️ .gitattributes             # Git LFS configuration

git clone https://github.com/A-01-hub/ISL.git
cd ISL

git lfs install
git lfs pull

python -m venv venv
venv\Scripts\activate

python3 -m venv venv
source venv/bin/activate

pip install --upgrade pip
pip install -r requirements.txt

python app.py

gunicorn --bind 0.0.0.0:7860 --workers 2 --threads 4 --timeout 120 app:app

# Build the Docker image
docker build -t sign-sarthi .

# Run the container
docker run -p 7860:7860 sign-sarthi

User signs: H → E → L → L → O
Display shows: "HELLO"

1. User signs: H → E → L → P (meant to type "HELLO")
   Display: "HELP"

2. User clicks "Delete Last" button
   Display: "HEL"

3. User signs: L → O
   Display: "HELLO"

4. User clicks "Add Space" button
   Display: "HELLO "

5. User signs: W → O → R → L → D
   Display: "HELLO WORLD"

6. User clicks "Speak" button
   Audio: "Hello World" (spoken by browser TTS)

1. Sign in ISL: H → I
   Display: "HI"

2. Select language: "Hindi"

3. Click "Translate"

4. Output: "नमस्ते" (Namaste)

{
  "frame": "data:image/jpeg;base64,/9j/4AAQSkZJRg..."
}

{
  "success": true,
  "prediction": "A",
  "confidence": 0.98,
  "processed_image": "data:image/jpeg;base64,..."
}

{
  "error": "No hands detected",
  "success": false
}

{
  "character": "A",
  "current_text": "HELL"
}

{
  "success": true,
  "predicted_text": "HELLA"
}

{
  "predicted_text": "HELLO WORLD"
}

{
  "success": true,
  "predicted_text": ""
}

{
  "predicted_text": "HELL"
}

{
  "predicted_text": "HELLO "
}

{
  "success": true,
  "text": "HELLO WORLD"
}

Dense(256, activation='relu') → Dropout(0.3) →
Dense(128, activation='relu') → Dropout(0.3) →
Dense(64, activation='relu') → Dropout(0.2)

0: WRIST
1-4: THUMB (CMC, MCP, IP, TIP)
5-8: INDEX (MCP, PIP, DIP, TIP)
9-12: MIDDLE (MCP, PIP, DIP, TIP)
13-16: RING (MCP, PIP, DIP, TIP)
17-20: PINKY (MCP, PIP, DIP, TIP)

[x0, y0, x1, y1, ..., x20, y20] = 42 features

# Development server (Flask built-in)
python app.py

# Production server (Gunicorn)
gunicorn --bind 0.0.0.0:7860 --workers 2 --threads 4 app:app

# Build image
docker build -t sign-sarthi:latest .

# Run container
docker run -d -p 7860:7860 --name sign-sarthi sign-sarthi:latest

# View logs
docker logs -f sign-sarthi

# Stop container
docker stop sign-sarthi

# Optional configurations
export PORT=7860
export PYTHONUNBUFFERED=1
export SECRET_KEY="your-secret-key"

Error: "Camera permission denied"
Solution:
1. Check browser permissions
2. Use HTTPS (required for camera on non-localhost)
3. Try different browser
4. Check if camera is being used by another application

Error: "Failed to load resource: model.h5"
Solution:
1. Ensure Git LFS is installed: git lfs install
2. Pull LFS files: git lfs pull
3. Verify model.h5 exists and is ~11.5 MB

Error: Characters vanish after adding
Solution:
1. Hard refresh browser (Ctrl+Shift+R)
2. Clear browser cache
3. Check console for JavaScript errors
4. Ensure latest code is deployed

Error: Wrong characters detected
Solution:
1. Improve lighting conditions
2. Position hand clearly in frame
3. Make distinct, clear gestures
4. Hold gesture for full 1 second
5. Avoid moving hand during detection

Error: Slow response time
Solution:
1. Close other browser tabs
2. Check internet connection
3. Reduce camera resolution
4. Use desktop instead of mobile

// Look for these log messages:
"Page loaded, starting camera..."
"Camera started successfully"
"Adding character: X (last: Y, cooldown: Zms)"
"Server response: {predicted_text: '...', success: true}"

# Install development dependencies
pip install -r requirements.txt
pip install black flake8 pytest

# Run linter
flake8 app.py

# Format code
black app.py

# Run tests (if available)
pytest tests/

MIT License

Copyright (c) 2024 Sign Sarthi Team

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.