A real-time eye tracking system based on Python, using webcam for gaze tracking, voice calibration, and real-time heatmap display. Designed for macOS with support for Chinese and English voice commands.
- English: README_EN.md - Complete English documentation
- ไธญๆ: README_CN.md - ๅฎๆดไธญๆ่ชชๆๆไปถ
This repository contains two eye tracking systems:
- Location:
gaze_heatmap/ - Features: ETH-XGaze model (default, best accuracy), L2CS-Net support, comprehensive CLI, advanced calibration
- Best for: Research, accuracy evaluation, detailed analysis
- Quick Start: See Gaze Heatmap Quick Start
- Location: Root directory (
gaze_tracker.py) - Features: Voice commands, simple interface
- Best for: Quick demos, simple use cases
- Quick Start: See Original System Quick Start
The new Gaze Heatmap System provides advanced gaze tracking with ETH-XGaze as the default model (best accuracy, supports MPS GPU acceleration):
Before using the system, please note:
- ๐ Viewing Distance: Maintain approximately 60cm distance from the screen for optimal accuracy
- ๐ฏ Head Position: Keep your head relatively still during calibration and tracking
- ๐ก Lighting: Ensure adequate and uniform lighting; avoid backlighting or shadows
- ๐ Glasses: Avoid reflective glasses that may interfere with tracking
- ๐ช Posture: Maintain a stable sitting posture throughout the session
- ๐ Recalibration: Recalibrate if you change position or lighting conditions significantly
cd gaze_heatmap
# Setup conda environment (one-time)
conda create -n gaze_eth python=3.10 -y
conda activate gaze_eth
pip install -r requirements.txt
# 1. Calibrate (required first)
python main.py calibrate --output my_calibration.yaml
# 2. Run live demo
python main.py demo --calibration my_calibration.yaml
# 3. Record session
python main.py record --calibration my_calibration.yaml --duration 60
# 4. Evaluate accuracy
python main.py evaluate --calibration my_calibration.yaml --num-points 20| Command | Description |
|---|---|
calibrate |
Run 9/16-point calibration procedure |
demo |
Live demo with real-time heatmap |
record |
Record gaze session with heatmap |
evaluate |
Evaluate tracking accuracy |
label |
Annotate recorded heatmaps |
- ๐ฏ ETH-XGaze Model: State-of-the-art gaze estimation (default, supports MPS GPU acceleration)
- ๐ Comprehensive CLI: Full command-line interface for all operations
- ๐ฌ Accuracy Evaluation: Built-in metrics (angular error, screen error, precision)
- ๐ Advanced Calibration: Polynomial regression with edge weighting
- ๐จ Real-time Heatmap: Live visualization with smoothing and fixation detection
- ๐พ Data Export: Save sessions, heatmaps, and evaluation reports
Based on testing results, the supported models rank as follows:
| Model | Accuracy | Performance | Notes |
|---|---|---|---|
| ETH-XGaze | โญโญโญ Relatively usable | Fast (MPS GPU support) | Default model, relatively better than alternatives |
| L2CS-Net | โญโญ Limited | Moderate | Requires manual weight download |
| MediaPipe | โญโญ Limited | Fastest | Fallback option, no additional setup required |
Test Results: ETH-XGaze > L2CS > MediaPipe
Note: All models have limitations in accuracy. ETH-XGaze performs relatively better among the available options and supports Apple Silicon GPU acceleration via MPS, making it the default choice.
Known Limitations:
โ ๏ธ Edge/Corner Sensitivity: All models show reduced sensitivity when gazing at screen edges and corners. The tracking accuracy decreases significantly in peripheral areas compared to the center of the screen.- This is a common limitation in webcam-based gaze tracking systems and may require additional calibration points at screen edges to improve edge accuracy.
For detailed documentation, see gaze_heatmap/how_to_run.md
# 1. Setup (one-time)
./setup.sh
# 2. Quick start
./quick_start.sh
# 3. High precision mode (best accuracy)
./run_high_precision.sh- ๐ฏ Real-time Gaze Tracking: High-precision facial feature extraction using Mediapipe
- ๐ค Voice Calibration: Support for "here"/"้่ฃก" voice commands for calibration and recording
- ๐ฅ Real-time Heatmap: Real-time gaze point heatmap display with smooth and decay effects
- ๐ Data Recording: Automatic recording of gaze point data to CSV files
- ๐ฅ๏ธ Multi-screen Support: Automatic detection of primary screen resolution
- ๐จ Visual Debugging: Optional camera debug window
- ๐ High Precision Mode: Multi-frame averaging, polynomial regression, edge weighting for improved accuracy
- ๐ง Camera Mirroring: Optional horizontal mirroring to fix left-right tracking issues
- ๐ Accuracy Evaluation: Built-in 5x5 test grid for quantitative accuracy assessment
- macOS 10.15+
- Python 3.8+
- Built-in or external webcam
- Microphone
- At least 4GB RAM
| Script | Description | Language |
|---|---|---|
./quick_start.sh |
Quick start with default settings | English |
./run_high_precision.sh |
High precision mode (best accuracy) | English |
./run_basic.sh |
Basic version with options | English |
./run_overlay.sh |
Advanced full-screen overlay version | English |
./run_high_sensitivity.sh |
High sensitivity version | English |
# Clone the repository
git clone <repository-url>
cd webcam_voice_label
# Run setup script
./setup.sh# Quick start (recommended)
./quick_start.sh
# High precision mode (best accuracy)
./run_high_precision.sh
# Basic version with debug window
./run_basic.sh
# Advanced full-screen overlay
./run_overlay.sh# Basic tracking
python gaze_tracker.py
# With debug window
python gaze_tracker.py --show-cam-debug
# High precision with camera mirroring
python gaze_tracker.py --rows 4 --cols 4 --cam-mirror
# Use Chinese model
python gaze_tracker.py --vosk-model ./vosk-model-small-cn-0.22
# Evaluate accuracy
python evaluate_accuracy.pyThe system generates a gaze_points.csv file with timestamped gaze coordinates:
timestamp,x,y
1640995200.123,960,540
1640995205.456,1200,300- High Precision Mode: ~1-2cm error (0.5-1ยฐ viewing angle)
- Ideal conditions: ~1-3cm error (1-2ยฐ viewing angle)
- General conditions: ~3-5cm error
- Difficult conditions: >5cm error
- Multi-frame Averaging: Collects 0.4 seconds of samples per calibration point
- Polynomial Regression: Uses 2nd-degree polynomial features for better non-linear mapping
- Edge Weighting: Gives higher weight to edge calibration points for better corner accuracy
- Camera Mirroring: Optional horizontal mirroring to fix left-right tracking issues
- Camera not working: Check System Preferences > Security & Privacy > Camera
- Microphone not working: Check System Preferences > Security & Privacy > Microphone
- Audio errors: Run
brew install portaudio && pip install --force-reinstall sounddevice - Poor tracking: Use high precision mode, recalibrate, check lighting, adjust posture, try camera mirroring
- Mediapipe: Facial feature extraction and iris detection
- Vosk: Offline speech recognition
- OpenCV: Image processing and display
- Scikit-learn: Regression model training
- PyQt6: Advanced overlay interface
This system uses polynomial regression rather than simple geometric calculations:
| Method | Accuracy | Why Better |
|---|---|---|
| Simple Geometry | 5-10cm error | Only works in ideal conditions |
| ML Regression | 1-3cm error | Handles real-world variations |
- Polynomial Regression: 2nd-degree features for non-linear mapping
- Multi-frame Averaging: 0.4s samples per calibration point for noise reduction
- Edge Weighting: Higher weight for corner calibration points
- Personal Adaptation: Learns individual facial geometry
f: R^n โ Rยฒ
f(facial_features) = (screen_x, screen_y)
Input: Normalized eye/iris coordinates
Output: Screen pixel coordinates
Model: Polynomial regression with edge weighting
For detailed technical documentation, see:
This project is licensed under the MIT License.
Welcome to submit Issues and Pull Requests to improve this project.
For questions or suggestions, please contact through GitHub Issues.