Control any game with your bare hands. No hardware. No controller. Just you.
[📹 Demo GIF / Video Placeholder] Hold both hands up in front of your webcam like gripping a steering wheel. Tilt to steer. Push apart to throttle. Bring together to brake.
┌─────────────────────────────────────────────┐
│ [ WEBCAM FEED PLACEHOLDER ] │
│ │
│ Left Hand ●────────────────● Right Hand │
│ ↕ Tilt = Steer │
│ ↔ Distance = Gas / Brake │
└─────────────────────────────────────────────┘
GHOSTHAND is an open-source, AI-powered gesture controller that turns your webcam into a full game input device — no physical controller required.
Using MediaPipe's 21-point hand landmark tracking averaged across both hands, GHOSTHAND computes a stable, noise-resistant hand center and maps real-world hand movements directly to virtual gamepad inputs via a software Xbox 360 controller.
Currently supports racing games as the first game profile, with FPS, shooter, and custom profiles in active development.
┌──────────────┐ ┌─────────────────────┐ ┌──────────────────────┐
│ │ │ │ │ │
│ WEBCAM │────▶│ MediaPipe Hands │────▶│ GHOSTHAND Engine │
│ (30fps) │ │ 21 Landmarks/Hand │ │ │
│ │ │ │ │ • Average 21 pts │
└──────────────┘ └─────────────────────┘ │ • Compute angle │
│ • Compute distance │
│ • Apply deadzone │
│ • Smooth output │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ vgamepad (Xbox360) │
│ │
│ Left Stick → Steer │
│ R Trigger → Gas │
│ L Trigger → Brake │
└──────────────────────┘
| Gesture | Input | Game Action |
|---|---|---|
| Tilt hands left/right (like a wheel) | Left Joystick X | Steering |
| Hands push apart (wider) | Right Trigger | Throttle / Gas |
| Hands pull together (closer) | Left Trigger | Brake |
| Calibration pose (flat, neutral) | Reference frame | Zero point |
Instead of tracking just the wrist (a single noisy point), GHOSTHAND averages all 21 MediaPipe hand landmarks to compute a stable hand center:
def get_hand_center(landmarks, w, h):
xs = [lm.x * w for lm in landmarks.landmark] # all 21 x coords
ys = [lm.y * h for lm in landmarks.landmark] # all 21 y coords
return int(np.mean(xs)), int(np.mean(ys)) # averaged centerThis means any one jittery point only contributes 1/21 to the result — giving rock-solid tracking even with partial occlusion or fast movement.
- Python 3.9+
- A webcam
- Windows OS (vgamepad requires Windows for virtual Xbox controller)
- A game that supports Xbox 360 controller input
# 1. Clone the repo
git clone https://github.com/yourusername/GHOSTHAND.git
cd GHOSTHAND
# 2. Create a virtual environment (recommended)
python -m venv venv
venv\Scripts\activate # Windows
# source venv/bin/activate # Mac/Linux
# 3. Install dependencies
pip install -r requirements.txtrequirements.txt
opencv-python>=4.8.0
mediapipe>=0.10.0
vgamepad>=0.1.0
numpy>=1.24.0
⚠️ vgamepad requires ViGEmBus Driver on Windows. Download and install from: https://github.com/nefarius/ViGEmBus/releases
python GHOSTHAND.py| Key | Action |
|---|---|
C |
Start 5-second calibration countdown |
Q |
Quit GHOSTHAND |
1. Launch GHOSTHAND
2. Hold both hands up in front of the camera
3. Position them like gripping a flat, level steering wheel
4. Press C — you have 5 seconds to hold the pose
5. The line between your hands turns GREEN = you're calibrated
6. Drive!
All tuning parameters are at the top of GHOSTHAND.py:
STEERING_SENSITIVITY = 1.0 # Multiplier on steering output
MAX_ANGLE_LOCK = 50 # Degrees of tilt for full lock
SMOOTHING_FACTOR = 0.5 # 0 = instant, 1 = never moves
DEADZONE_STEER = 1500 # Joystick units ignored near center
DIST_RANGE = 120 # Pixel range for full throttle/brake
DEADZONE_DIST = 40 # Pixel deadzone for gas/brake
STEER_INVERT = 1 # Set to -1 to flip steering direction| Game | Profile | Status |
|---|---|---|
| Forza Horizon 5 | Racing | ✅ Working |
| BeamNG.drive | Racing | ✅ Working |
| Need for Speed Heat | Racing | ✅ Working |
| Rocket League | Racing/Sports | 🧪 Experimental |
GHOSTHAND is just getting started. Here's what's coming:
A full profile switcher letting you load different gesture → input mappings per game genre.
GHOSTHAND/
profiles/
racing.json ← current
fps.json ← coming soon
shooter.json ← coming soon
fighting.json ← planned
Map hand gestures to mouse + keyboard inputs for first person games:
- Aim — track dominant hand position to move crosshair
- Shoot — pinch gesture (index + thumb) = left click / fire
- ADS (aim down sights) — bring hands close together
- Reload — specific fist + open gesture combo
- Movement — off-hand tilt controls WASD equivalent
Replace the raw script with a proper application:
- Game profile selector with live preview
- Sensitivity sliders with real-time feedback
- Auto-detection of running games
- Save / load custom profiles
- Built with PyQt6 or Electron
Train a custom classification model (TFLite / ONNX) on top of MediaPipe landmarks to recognize complex gestures:
- Fist = handbrake
- Peace sign = horn / special action
- Open palm = pause / menu
- Thumbs up = confirm
- Linux / Mac support via alternative virtual controller drivers
- WebSocket bridge for browser-based games
- Mobile companion app (phone as secondary controller)
Official profiles and testing for:
| Genre | Target Games |
|---|---|
| Racing | Forza, NFS, Gran Turismo |
| FPS | CS2, Valorant, Apex Legends |
| Third-Person Shooter | GTA V, The Division |
| Fighting | Mortal Kombat, Street Fighter |
| Sports | FIFA, Rocket League |
| Battle Royale | PUBG, Fortnite |
GHOSTHAND/
├── GHOSTHAND.py ← Main script (current prototype)
├── requirements.txt
├── README.md
└── profiles/ ← (coming v2.0)
├── racing.json
└── fps.json
GHOSTHAND is built for the open source community. Contributions are welcome.
# Fork → Clone → Branch → PR
git checkout -b feature/fps-profile
git commit -m "feat: add FPS shooter gesture profile"
git push origin feature/fps-profileAreas we especially need help with:
- Testing with more games and reporting results
- Building out new game profiles
- Improving gesture recognition accuracy
- Cross-platform support (Linux / Mac)
- UI/UX for the desktop app (v3.0)
MIT License — free to use, modify, and distribute.
- MediaPipe — Hand landmark detection
- OpenCV — Camera capture & visualization
- vgamepad — Virtual Xbox 360 controller
- NumPy — Landmark averaging math
GHOSTHAND — because the best controller is the one you can't see.