Repository navigation
3.41.0
Audio-driven lipsync
Speech in, ARKit blendshape animation out — NVIDIA Audio2Face-3D v2.3, running locally through ONNX. Drives any mesh that has ARKit morph targets, including the ones qtmesh facerig generates.
qtmesh lipsync take.wav --mesh head.glb -o spoken.glb
qtmesh lipsync take.wav --mesh head.glb --fps 60 --clip Speech -o out.glb
qtmesh lipsync take.wav --mesh head.glb --emotion joy=0.6 -o out.glbIn the editor it sits in the Inspector directly beneath Add ARKit Blendshapes — rig the face, then drive it. The solve runs on a worker thread with progress and cancel, and a whole take commits as one Ctrl+Z.
How it works
A2F doesn't emit blendshapes; it emits a deformation of NVIDIA's own character mesh. What makes that usable is that NVIDIA also ships that character's ARKit-52 blendshape deltas, so the weights can be recovered by asking which combination of those 52 shapes best reproduces the predicted motion. The answer is a vector of weights, and weights carry no topology — so no correspondence between their head and yours is ever needed. 51 of our 52 names match theirs verbatim.
WAV → mono → 16 kHz → centred window per frame
→ network.onnx (+ 26-dim emotion) → 301 coefficients
→ PCA reconstruct → constrained solve against the ARKit-52 basis
→ morph-weight keyframes
No new dependencies. Both .npz archives are STORED, so the reader walks the zip directory directly — no zlib. The WAV reader walks RIFF chunks properly rather than assuming a 44-byte header.
Two things that make it look right
mouthClose is a counter-shape. ARKit defines it as a partial cancellation of jawOpen, so on a correct rig it lifts the lower lip hard on its own — which is why mouthClose alone pushed the lip to the nose while jawOpen 1.0 + mouthClose 1.0 looked fine. NVIDIA's character uses the opposite convention (cosine with jawOpen +0.41 vs the ICT rig's −0.46), so the solved weight over-drives on replay. Corrected through NVIDIA's own multiplier table: peak weight halved, and frames where mouthClose ran at more than twice jawOpen went 9 → 0.
Keyframes carry every channel. ARKit targets share one VAT_POSE track, and Ogre interpolates a pose missing from the next keyframe toward zero. Skipping unchanged channels therefore made steady channels dip to 0 and snap back — measured at 39 dips across 14 channels, jawOpen going 0.056 → 0.000 → 0.080 between adjacent frames. Now 0.
Licensing
NVIDIA Open Model License: commercial use and redistribution permitted, NOTICE file required — written beside the weights on first fetch. Models download from NVIDIA's own repo rather than a mirror, with their published SHA-256 digests pinned, so a corrupted download is detected and re-fetched instead of silently producing garbage. Audio2Emotion is separately licensed ("use allowed with Audio2Face only") and is deliberately not shipped; emotion is a user-supplied vector instead.