Real-time, hands-free desktop control pipeline empowering users with motor impairments to interact with operating systems via head pose estimation and trainable micro-gestures.
Python Computer Vision Inference Latency License: MIT
Traditional Human-Computer Interaction (HCI) devices—mice, trackpads, and physical keyboards—assume fine motor control. For individuals living with ALS, cerebral palsy, spinal injuries, or severe tremors, standard input devices present substantial barriers.
TrinetraAI bridges this gap by turning any consumer-grade 30 FPS webcam into an input interface:
Zero Specialty Hardware: Runs locally on commodity CPU hardware without specialized eye-trackers.
Low-Latency Loop: End-to-end inference-to-dispatch loop runs at ~28ms (p95 ~38ms), well within interactive thresholds (<50ms).
Adaptive Calibration: Combines geometric head-pose heuristics with user-trainable gesture classifiers (SVM/LSTM) to prevent fatigue and accommodate individual physical ranges.
flowchart TD subgraph Vision Pipeline ["Threaded Vision Ingestion (~1 ms)"] Cam[Webcam Stream @ 30 FPS] --> Capture[WebcamCapture Thread] Capture --> BGR[BGR Frame Buffer] end
subgraph Feature Extraction ["Landmark & Geometry Extraction (~22 ms)"]
BGR --> FaceTracker[FaceMesh Tracker]
BGR --> GestureDetector[Hand Gesture Detector]
FaceTracker --> Angles[Pose Estimation: Yaw / Pitch / Roll]
GestureDetector --> Vec63[63-d Landmark Coordinate Vector]
end
subgraph Decision Engine ["Inference & Mapping (~2 ms)"]
Angles --> Heuristics{Angle Thresholds}
Vec63 --> Classifier[Model: Linear SVM / Dynamic LSTM]
Heuristics --> ActionMapper[Action Dispatcher & Debouncer]
Classifier --> ActionMapper
end
subgraph OS Execution ["Input Synthesis (<1 ms)"]
ActionMapper --> NativeOS[pynput / PyAutoGUI Native Primitives]
ActionMapper --> HUD[cv2 HUD Overlay: FPS & Latency Readout]
end
Metrics benchmarked on a standard quad-core Intel i5 laptop CPU @ 640×480 resolution:
Metric Measured Value Target Budget Status End-to-End Latency ~28 ms < 50 ms Optimal p95 Latency ~38 ms < 60 ms Optimal Gesture Accuracy (Held-out) ~95.2% > 90% High Confidence False-Positive Trigger Rate < 2.8% < 5% Stable Frame Dropping Rate 0.0% < 1% Threaded I/O
- Head Pose Controls (Continuous Tracking)
Gesture / Pose Trigger Threshold OS Dispatched Action Head Turn Left yaw > +15° Smooth scroll window left Head Turn Right yaw < -15° Smooth scroll window right Head Nod Down pitch > +10° Scroll down Head Tilt Up pitch < -10° Scroll up
- Hand Micro-Gestures (User-Trained Classification)
Gesture Class Model Feature Vector Default OS Action neutral 63-d Cartesian normalized coordinates Idle / No Action open_palm 21-point hand mesh topology Primary Mouse Click (Left) fist Closed finger joints Secondary Click (Right) peace_sign Extended index + middle fingers Configurable App Launcher
TrinetraAI/ ├── config/ │ ├── settings.yaml # Thresholds, debounce windows, and key bindings │ └── loader.py # YAML to typed frozen dataclasses ├── vision/ │ ├── capture.py # Dedicated capture thread yielding BGR frames │ ├── face_tracker.py # MediaPipe FaceMesh -> head-pose yaw/pitch/roll │ ├── gesture_detector.py # MediaPipe Hands -> 63-d normalized landmark vectors │ └── overlay.py # cv2 heads-up display (landmarks, labels, latency) ├── classifier/ │ ├── gesture_model.py # Inference runtime (Linear SVM & PyTorch LSTM) │ └── trainer.py # CLI utility for sample recording and model fitting ├── control/ │ ├── actions.py # PyAutoGUI & pynput automation primitives │ └── action_mapper.py # Confidence gating, debounce timers, action dispatch ├── tests/ # Unit tests for capture, geometry, and config ├── requirements.txt └── main.py # Orchestrator and event loop
Prerequisites
Python 3.10 or higher
Standard RGB webcam (720p @ 30 FPS recommended)
OS: Linux (X11 / Wayland), macOS, or Windows 10/11
- Clone & Set Up Environment
git clone https://github.com/chetx27/TrinetraAI.git cd TrinetraAI
python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate
pip install --upgrade pip pip install -r requirements.txt
- Run with Defaults
Launch the assistant using the pre-trained SVM model:
python main.py --config config/settings.yaml
Press q or ESC in the HUD window to exit and print session latency metrics.
🎯 Calibrating & Training Custom Gestures
If standard gesture boundaries do not suit your range of motion, record your own calibration set:
python -m classifier.trainer --gesture neutral --samples 60 python -m classifier.trainer --gesture open_palm --samples 60 python -m classifier.trainer --gesture fist --samples 60 python -m classifier.trainer --gesture peace_sign --samples 60
python -m classifier.trainer --fit
python -m classifier.trainer --fit --lstm python main.py --config config/settings.yaml --lstm
Tune interaction thresholds directly in config/settings.yaml:
vision: camera_index: 0 frame_width: 640 frame_height: 480 fps: 30
control: debounce_time_ms: 350 scroll_speed: 15 head_pose: yaw_threshold: 15.0 pitch_threshold: 10.0
model: backend: "svm" # Options: "svm", "lstm" confidence_cutoff: 0.85
Contributions make open-source assistive tools better for everyone.
Fork the Project.
Create your Feature Branch (git checkout -b feature/dynamic-smoothing).
Commit your Changes (git commit -m "feat: implement Kalman filter for head pose").
Push to the Branch (git push origin feature/dynamic-smoothing).
Open a Pull Request.
Please check open issues tagged good first issue to pick up starter tasks.
┌───────────┐ ┌──────────────────┐ ┌────────────┐ ┌────────────┐
│ Webcam │───▶│ MediaPipe │───▶│ Classifier │───▶│ OS Action │
│ 30 fps │ │ FaceMesh/Hands │ │ SVM / LSTM │ │ Scroll, │
│ 640×480 │ │ Landmarks │ │ 8 classes │ │ Click, Key │
└───────────┘ └──────────────────┘ └────────────┘ └────────────┘
│ │ │ │
│ ┌────────▼────────┐ │ │
│ │ Head Pose │ │ │
│ │ yaw/pitch/roll │────────────┘ │
│ │ (threshold) │ │
│ └─────────────────┘ │
│ │
└──────────────────┐ │
┌────────▼────────┐ │
│ Overlay │◀─────────────────────────────┘
│ landmarks, label│ (feedback)
│ latency HUD │
└─────────────────┘
pip install -r requirements.txt
python main.py --config config/settings.yamlPreview-only mode (no OS actions fired):
python main.py --config config/settings.yaml --no-actionsRecord each hand-gesture class (head gestures use thresholds, no training needed):
python -m classifier.trainer --gesture neutral --samples 60
python -m classifier.trainer --gesture open_palm --samples 60
python -m classifier.trainer --gesture fist --samples 60
python -m classifier.trainer --gesture peace_sign --samples 60Or fit after recording all classes:
python -m classifier.trainer --fitTo use the LSTM backend instead of SVM:
python -m classifier.trainer --fit --lstm
python main.py --config config/settings.yaml --lstm| # | Gesture | Trigger Condition | OS Action |
|---|---|---|---|
| 1 | head_turn_left |
yaw > +15° | Scroll window left |
| 2 | head_turn_right |
yaw < −15° | Scroll window right |
| 3 | head_nod_down |
pitch > +10° | Scroll down |
| 4 | head_tilt_up |
pitch < −10° | Scroll up |
| 5 | open_palm |
Classifier (SVM/LSTM) | Left click |
| 6 | fist |
Classifier (SVM/LSTM) | Right click |
| 7 | peace_sign |
Classifier (SVM/LSTM) | App launcher (configurable) |
| 8 | neutral |
Default / low conf. | No action |
| Module | Responsibility | Key Class | Latency Budget |
|---|---|---|---|
vision/capture.py |
Threaded webcam I/O | WebcamCapture |
~1 ms |
vision/face_tracker.py |
FaceMesh → head-pose angles | FaceTracker |
~12 ms |
vision/gesture_detector.py |
Hands → 63-d landmark vector | GestureDetector |
~10 ms |
vision/overlay.py |
Draw HUD (landmarks, label, latency) | Overlay |
~1 ms |
classifier/gesture_model.py |
SVM/LSTM classification | GestureModel |
~1 ms |
classifier/trainer.py |
Record samples, fit model | CLI | offline |
control/actions.py |
pynput/pyautogui OS primitives | functions | <1 ms |
control/action_mapper.py |
Label → action dispatch | ActionMapper |
<1 ms |
config/loader.py |
YAML → typed frozen dataclasses | AppConfig |
startup only |
utils/metrics.py |
Ring-buffer latency, accuracy stats | LatencyTracker |
<0.01 ms |
main.py |
Event-loop wiring, CLI entry point | run() |
orchestrator |
| Metric | Value |
|---|---|
| E2E latency | ~28 ms |
| p95 latency | ~38 ms |
| Gesture acc. | ~95 % |
| False-action | <3 % |
Note: Metrics measured on an Intel i5-10th-gen laptop CPU @ 640×480. Actual values depend on hardware and lighting conditions. Run the assistant and check the session stats printed on exit.
main.run() 28.0 ms ████████████████████████████
├── WebcamCapture.read() 0.8 ms █
├── FaceTracker.process() 12.1 ms ████████████
│ ├── cv2.cvtColor() 0.3 ms ▏
│ ├── FaceMesh.process() 10.5 ms ██████████▌
│ └── cv2.solvePnP() 1.2 ms █▏
├── GestureDetector.process() 9.8 ms █████████▊
│ ├── cv2.cvtColor() 0.3 ms ▏
│ ├── Hands.process() 9.0 ms █████████
│ └── normalise + flatten 0.4 ms ▍
├── GestureModel.predict() 0.6 ms ▌
│ ├── StandardScaler.transform() 0.1 ms ▏
│ └── SVC.predict_proba() 0.5 ms ▌
├── ActionMapper.execute() 0.2 ms ▏
├── Overlay.draw() 0.9 ms █
│ ├── draw_landmarks() 0.5 ms ▌
│ ├── draw_label() 0.2 ms ▏
│ └── draw_latency() 0.2 ms ▏
└── cv2.imshow() + waitKey() 3.6 ms ███▌
ambient-vision/
├── vision/
│ ├── capture.py # OpenCV webcam thread; yields BGR frames at 30 fps
│ ├── face_tracker.py # MediaPipe FaceMesh → yaw/pitch/roll angles
│ ├── gesture_detector.py # MediaPipe Hands → 21-landmark vector per hand
│ └── overlay.py # cv2 HUD: draw landmarks, current label, latency
├── classifier/
│ ├── gesture_model.py # scikit-learn SVM (default) OR Keras LSTM (--lstm)
│ ├── trainer.py # CLI: record N samples/class, fit, save model.pkl
│ └── data/ # .npy feature arrays, auto-created by trainer.py
├── control/
│ ├── action_mapper.py # dict: label → callable OS action
│ └── actions.py # pynput + pyautogui primitives: scroll, click, hotkey
├── config/
│ ├── settings.yaml # all thresholds, gesture→action map, camera index
│ └── loader.py # typed dataclass from yaml; fail-fast on bad keys
├── utils/
│ └── metrics.py # RingBuffer latency tracker; accuracy helpers
├── tests/ # pytest suite — one file per module
├── main.py # entrypoint; argparse; wires all modules
├── requirements.txt # pinned versions
├── Makefile # targets: install, run, train, test, lint
└── README.md
# Install dependencies
make install
# Run the assistant
make run
# Train all gestures
make train
# Run tests
make test
# Lint + type check
make lint
# Auto-format
make format- MediaPipe — Google's ML framework for face and hand tracking
- OpenCV — Computer vision library
- scikit-learn — Machine learning (SVM classifier)
- pynput / PyAutoGUI — OS input control
- Head-pose estimation approach adapted from learnopencv.com