An Implementation of VTuber (Both 3D and Live2D) using Python and Unity. Supporting face movement tracking, eye blinking detection, iris detection and tracking and mouth movement tracking using CPU only.
Live2D Demo
UnityChan 3D Demo
- Facial landmarks detection and movement tracking supported by Facemesh by Mediapipe.
- Various facial expressions detection including eye blinking, iris and mouth movements.
- Running smooth 30 FPS with CPU only for the aformentioned features.
- Simple and clean UI for adjusting the sensibility of detection in Unity.
- Saveload mechanism to save and load your preferences in Unity.
- Including sample (Unity) projects for both 3D and Live2D models.
- Support Window and M1 machines.
- Detailed and thorough explanation videos (with EN and ZH subtitles) playlist
| File | Description |
|---|---|
| main.py | The main program |
| facial_landmark.py | The module which is used to detect your face and generate the facial landmarks. |
| pose_estimator.py | The module which estimates your pose/ orientation of your head based on the landmarks. |
| stabilizer.py | Implementation of Kalman Filter to stabilize the values. |
| facial_features.py | Various facial features detection implementation, including blinking, iris detection and mouth movement. |
| model.txt | The points of the 3D Canonical model used in Mediapipe. Source file |
| UnityAssets | Whole Unity Projects (in packages) and Scripts (old version) for both 3D (UnityChan) and Live2D (Hiyori) models |
Using avatars for streaming, content creation and VR gaming has been gaining increasing popularity, especially the boom of Hololive and other related companies active apperances in social media platforms such as YouTube and Twitter. Curious about the technology behind, I create this project after multiple researches.
Existing projects rely on Dlib, which although providing reliable and accurate facial landmark detection, requires decent graphic cards to run. However, implemented with the recent FaceMesh model in Mediapipe, accurate detection and tracking can be run smoothly using CPU only, making running on computers with mediocre graphic cards or laptops with integrated graphic cards possible.
Clone this project into your directory
git clone https://github.com/mmmmmm44/VTuber-Python-Unity.git
cd "VTuber-Python-Unity"
-
Create An empty Unity 3D Project
-
Import either the Live2D or UnityChan3D package to your project. The corresponding SDKs have been included already. (Excpet that I removed the voice files of UnityChan in the UnityChan Unity package due to file size limit in github.)
- last edit: 16-05-2022
-
[NEW] In the Game window of Unity, create a 9:16 portrait ratio display mode and save it. Then the UI will return normal. (Or you can adjust the ratio in the AspectGridLayoutCellSize.cs attached on each of the panel Game Object, namely "Setting Panel" and "TCP Setting Panel".) Tutorial
-
Run the Scene. Click the Setting button to switch on the TCP server. There should be a "Waiting for connection..." log message showing in the Console of Unity.
-
Run the following code in terminal [content in the bracket is optional]
python main.py --connect [--debug] [--port PORT]
- Enjoy
Video Walkthrough: Click Me!
-
Download the Cubism SDK For Unity from this website and the sample model used (桃瀬ひより) from this website
-
Create an empty Unity 3D project, and import the Cubism SDK. Unzip the model and drag the whole folder to the Project window of the Unity Project.
-
Drag the live2D model's prefab into the scene. Run the scene immediately to allow the model to be shown in Scene and Game window.
-
Adjust the camera's position, background and projection properties. If there are some werid projection problems of the model, changing the projection of the camera from Perspective and Orthographic works for me.
-
Drag the HiyoriController.cs to the Hiyori GameObject. Adjust the parameters in the inspector
-
Run the scene.
-
Run the following code in terminal [content in the bracket is optional]
python main.py --connect [--debug] [--port PORT]
- Enjoy
Video Walkthrough: Click Me!
-
Download the UnityChan model from the website. Go to "Data Download", accept the terms and agreements, and select the first one. Unzip the file.
-
Create an empty Unity 3D Project. Drag the unzipped folder to the Project Window of the project.
-
Go to UnityChan\Prefabs and Drag the "unitychan" prefab into the scene.
-
Adjust the camera's position, background and field of view.
-
Drag the UnityChanControl.cs script onto the prefab. Change the update mode of the Animator attached to "Animate Physics" and the Controller to UnityChanLocomotions. (Crucial) Adjust the variables in the inspector. Disable other attached scripts except AutoBlink and UnityChanController. You may tick the box "is Auto Blink Active" in UnityChanContoller to enable auto blinking (enable AutoBlink script when ticked).
-
Run the scene first
-
Run the following code in terminal [content in the bracket is optional]
python main.py --connect [--debug] [--port PORT]
- Enjoy
Make sure you run the Unity Scene first before running the python script
The complete Unity Project with fancy stuffs such as UI system, Save Load Preferences, and custom port UI can be found after importing the Unity Packages provided.
-h, --help show this help message and exit
--connect connect to the unity character
--port PORT specify the port of the connection to
unity. Have to be the same as in Unity.
--cam CAM specify the camera number to use
if you have multiple cameras connected
to the computer.
(for cv2.VideoCapture(CAM))
--debug show the raw values of the detection
in the terminal
Connecting to the Unity Project with default port, showing the sent data.
python main.py --connect --debug
Connecting to the Unity Project with custom port (e.g. 5077), showing the sent data.
python main.py --connect --debug --port 5077
-
Python 3.8.5
-
Numpy 1.19.2
-
OpenCV 4.5.1
-
Mediapipe 0.8.9.1
-
Unity 2020.3.12f1
(Later version should be supported as well)
(For Windows, it is recommended to run this project using Anaconda and create a virtual environment before installing such packages.)
The whole project is run on a laptop with Intel Core i5-8250U, with 16GB RAM and integrated graphic card only.
The same project is tested on a M1 Max device, running in rosetta.
Eye motion tracking - Opencv with Python - Pysoruce
| Project | Author | LICENSE |
|---|---|---|
| head-pose-estimation | Yin Guobing | LICENSE |
| VTuber_Unity | AI葵 | LICENSE |
| VTuber-MomoseHiyori | KennardWang | LICENSE |
Hiyori Momose's model
| Position | Creator |
|---|---|
| Illustration | Kani Biimu [Twitter @kani_biimu] |
| Modeling | Live2D Inc. |
MIT
The Unity Chan model in the Unity Packages provided is distributed under Unity-Chan License © Unity Technologies Japan/UCL. A seperate sets of that License is included in UnityAssets/Licenses/UCL2_0