-
Notifications
You must be signed in to change notification settings - Fork 70
Mediapipe Webcam Tracker
In addition to tracking using OpenSeeFace and the NVIDIA RTX webcam tracker, VTube Studio now supports high-quality webcam tracking using the new Google Mediapipe Webcam tracker. The quality is roughly comparable to the NVIDIA tracker but no special hardware is required.
It is directly included in VTube Studio (Windows only for now), no DLC or download needed.
The Google Mediapipe tracker is only available for Windows, but unlike the NVIDIA tracker, no special GPU is required. The tracker is directly included in VTube Studio, no further download or DLC needed.
The Google Mediapipe tracker can be started just like the regular OpenSeeFace webcam tracker.
Just select "[Google] Mediapipe Tracker" after clicking the "Tracking Quality" button. This will only be available on Windows.
After starting the tracker for the first time, make sure to calibrate at least once while making a neutral face and looking straight at the screen. VTube Studio remembers the calibration data so you don't have to do that every time you start the tracker or restart VTube Studio.
In terms of tracking quality, I would say it is pretty good but not quite as good as the NVIDIA tracking. With further updates, it could definitely get as good, if not better. I will also look into bringing this to the Android version of VTube Studio.
Mouth-tracking is pretty accurate and so is eye-tracking. Blink-tracking works well. Wink-tracking is fine too, but as always, depending on your eye shape/size it may work better or worse. You'll just have to try it out and see for yourself.
The face rotation range of the Google Mediapipe tracker is a bit more restrictive than most other trackers. If you look too much down or left/right, it may lose tracking.
It supports the same parameters you have with iOS tracking, including "Mouth X" and individual brow tracking, but it DOES NOT currently support cheek puff and tongue-tracking. The tracker is in active development, so it is likely that those will be added eventually.
It should be comparable to the OpenSeeFace webcam tracking in terms of CPU/GPU usage.
Hand tracking is supported when using the Google Mediapipe tracker.
I've tried it with glasses and the eye-tracking and wink-tracking seems to work fairly well. Of course, it depends on the webcam placement and lighting too.
Of course! It is available and will always stay available as the main webcam tracker. This is just another option you can try out.
Parameter ranges/setups will be as close to iOS tracking as I can get them, so all your existing models should work without any changes (except cheek-puff and tongue-tracking since they are not available with this tracker for now).
If you have any questions that this manual doesn't answer, please ask in the VTube Studio Discord !!
- Introduction & Requirements
- Known Issues
- Where to get models?
- Android vs. iPhone vs. Webcam
- Getting Started
- Loading your own Models
- Loading your own Backgrounds
- VTube Studio Settings
- Display Light Overlay
- Visual Effects
- Spout2 Background
- VTS Model Settings
- Expressions ("Stickers"/"Emotes")
- Animations
- Interaction between Animations, Tracking, Physics, etc.
- Streaming to Mac/PC
- Recording/Streaming with OBS
- Starting without Steam
- Starting as Admin
- Restore old VTS Versions
- NVIDIA Broadcast Face Tracker
- Google Mediapipe Face Tracker
- Tobii Eye-Tracker
- Hand-Tracking
- Lipsync
- Controlling multiple models with one Webcam or iPhone/Android device
- Connection Issues & Troubleshooting
- Taking/Sharing Screenshots
- Item System
- Live2D-Items
- Recoloring Models and Items
- Record Animations
- Multiplayer
- VNet Security
- Twitch Interaction
- Add Special ArtMesh Functionality
- Item Scenes & Item Hotkeys
- Plugins (YouTube, Twitch, etc.)
- VTube Studio Model File
- Copy config between models
- Sending data to VSeeFace
- Steam Workshop
- How to fix lag issues
- How to fix webcam issues
- Live2D Cubism Editor Communication
- FAQ
- Privacy Policy, Licensing and Further Terms