Transform real-time body movements — including walking, squatting, hand swinging, and head tilting — into keyboard inputs, enabling a more interactive and intuitive way to control apps and games.
Control an RPG game (Legend of Zelda) (old version)
Control a racing game (old version)
Install python3
Create an virtual environment (optional)
python -m venv venv
.\venv\Scripts\activateInstall packages
pip install -r requirements.txtpython app.pypython -m nuitka --disable-console --product-name MotionMap app.pyRun the build file in the root directory
- Windows:
app.exe
Movements are based on Mediapipe Pose model. The model detects 33 key points of the body, including eyes, ears, nose, shoulders, elbows, wrists, hips, knees, and ankles. From these key points, we can detect the movements by calculating the angles between the points, the distance between the points, and the position of the points.
See details in file src/movements.py
Tilt head in left/right
- Swing hands
- Left: swing hand from left to right
- Full swing: swing left hand from head to bottom
- Raise both hands up
- Hands crossed: Cross 2 hands in Wakanda style
- Walking:
- Up: walking with 2 hands down
- Left: walking with only left hand up 90 degree
- Right: walking with only right hand up 90 degree
- Backwards: : walking with both hands up 90 degree
- Squat
- Left/right leg raised
- Move 2 hands close to enable steering wheel, tilt left or right to control
- Move 2 hands inside the green box to enable driving up control
- Support more modes: driving, FPS games, controlling mouse
- Support 2 players mode
- Support more OS: Mac, Linux
- Edit movements detection thresholds (advanced mode)
- Fix the mirror camera issue
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.