A machine learning approach for tracking face expressions and transferring them to Unity3D and mixamo characters
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
UnityFiles
co
pose
pts
README.md
aggregatePose.m
aggregatePts.m
demo.m
demoimg.png
demoimg2.png
nets.mat
tris.mat
unitysetup.png

README.md

mixamoFaceTransfer

A machine learning approach for tracking face expressions and transferring them to Unity3D and mixamo characters

In order to run the code, you will need to use matlab to generate the blendshape coefficients. The demo.m will generate the txt files shown in the folder “co”.

To generate the facial points (files located in pts), we used the DEST opensource software. The dest_align command will compute the 68 key points per face. The pose (files located in pose) were generated by openface. FaceLandmarkImg with the gaze parameter will generate the pose files. Samples are shown in pts and pose folders here.

Copy the coefficients into a Unity directory called “coeffs”. Import the “test.UnityPackage” to get the Mixamo model. View the image for the proper setup of the model and scripts.

alt text

You can see a sample by copying the text files from “co” into the “coeffs” directory.

“mixamo”

If you use this software, please cite

Edward Kim, Christopher Moritz, “Enhancing the Communication Spectrum in Collaborative Virtual Environments”, 12th International Symposium on Visual Computing, ISVC 2016.