Submission for ML Empowerment Build Challenge 2.0: Learn AI (Devpost)
Bridging the communication gap for Deaf and hard-of-hearing students in inclusive classrooms through AI Computer Vision, Live Captions, Groq AI Summaries, and a Gamified Quiz Arena.
In traditional inclusive classrooms, Deaf and hard-of-hearing (DHH) students can face communication barriers. Human sign language interpreters are not always available for every class, while traditional speech-to-text systems do not always provide enough interaction for students to actively participate.
Because of this, DHH students may have difficulty:
- Following spoken teacher instruction in real time.
- Signaling questions or needs during a lesson.
- Participating in classroom activities.
- Reviewing and testing their understanding after the lesson.
We wanted to build a solution that helps students communicate, follow lessons, and stay engaged using devices that classrooms already have.
SignBridge AI is a web application designed to make inclusive classrooms more interactive and accessible for Deaf and hard-of-hearing students.
It combines real-time hand gesture detection, live captions, AI lesson summaries, and a gamified quiz system into one platform.
SignBridge AI uses Google MediaPipe Hands to detect 15 distinct ASL/classroom gestures directly in the browser.
Examples include:
- Raise Hand
- Need Help
- Repeat Please
- Understand
- Not Feeling Well
- and other classroom-related gestures.
When a gesture is detected, the teacher receives a visual notification so the student can communicate without interrupting the lesson verbally.
Teachers can speak normally while SignBridge AI converts their speech into live captions.
The system supports:
- English
- Indonesian
- Continuous speech recognition
- Interim results for low-latency captions
- Automatic capitalization
- Spoken punctuation handling
- Fullscreen projector mode
The speech recognition uses the browser's Web Speech API, so it does not require a separate paid speech-to-text API.
After a lesson, SignBridge AI can turn the lesson transcript into an interactive quiz.
The system generates 10 multiple-choice questions based on the lesson transcript using Groq AI with openai/gpt-oss-120b.
The Quiz Arena includes:
- Score tracking
- Correct and wrong answer counters
- Combo streaks
- Progress bar
- Answer explanations
- Gesture-based answer controls
- Dynamic score-based celebrations
- Confetti animations
- Game-show style background music
- Victory fanfare and sound effects
Long lesson transcripts can be difficult for students to review.
SignBridge AI uses Groq AI to generate structured lesson summaries containing:
- Main topics
- Important concepts
- Key points
- Useful takeaways
This gives students a simpler way to review what was discussed in class.
SignBridge AI allows classroom data to be synchronized between teacher and student devices.