Hackathon Project done at Cheesehacks
Gesture is a software that utilizes hand gesture detection using Mediapipe, which allows users to interact with their screen purely through the use of their hand signs. Users are capable of configuring their own options for what interaction each hand gesture will make. We utilized portions of the code from this original repo. Also some extra thanks to this repo for translating to english.
Made by @ahaanbohra @bhinu @B-a-1-a @junw000
Traditional input methods, such as keyboards and mice, can be limiting, especially in scenarios that demand hands-free interaction, greater accessibility, or enhanced efficiency. Gesture addresses these challenges by enabling users to control their systems using hand gestures, which are:
- Intuitive: Natural movements for seamless interaction.
- Efficient: Automates repetitive tasks in a fraction of the time.
- Customizable: Users can assign their own commands to gestures.
- Accessible: Designed to assist users with physical disabilities or those in environments where traditional input is impractical.
- Gesture Recognition: Accurate real-time hand tracking and gesture classification powered by OpenCV and MediaPipe.
- Custom Command Mapping: Assign personalized commands to gestures, such as:
- Taking a screenshot with a simple hand sign.
- Opening a browser with a wave.
- Locking the screen with a gesture.
- Advanced AI Integration: CNNs trained with TensorFlow ensure robust gesture recognition, even in challenging environments.
- Cross-Platform Support: An Electron-based desktop app that works on Windows, macOS, and Linux.
The system operates using a multi-stage pipeline:
- Input images are processed using OpenCV and passed through a hand detection neural network.
- Detected hand landmarks are fed into a convolutional neural network (CNN) for feature extraction.
- A feed-forward neural network classifies the gestures into predefined or user-defined categories.
The flowchart below illustrates the pipeline:
- OpenCV: For real-time video capture and preprocessing.
- MediaPipe: For hand landmark detection and tracking.
- TensorFlow: For training and deploying CNNs for gesture classification.
- Electron: To build a cross-platform desktop application.
To get started with Gesture, follow these steps:
# Clone this repository:
git clone https://github.com/junw000/Gesture.git
# Install dependencies:
cd Gesture
pip install -r ./backend/requirements.txt
npm install
# Start the application:
npm start