EchoFace

EchoFace

EchoFace is SEE's real-time facial animation module driven by webcam-based face tracking. It controls an avatar's face using blendshapes, head rotation, and eye rotation, keeping the animation synchronized across all clients in a multiplayer session.

Overview

A brief overview of EchoFace:

• Captures the local player's face via a webcam, using Google MediaPipe's Face Landmarker task.
• Applies the tracked expression to the player's own avatar's SkinnedMeshRenderer and bone Transforms (mouth, eyes, brows, jaw, head pose, and eye-look direction).
• Synchronizes this data across the network, so every client sees the same facial animation on that avatar.
• Can be toggled on and off at runtime with a single key press, and automatically avoids conflicts with other facial-animation modules already present on the avatar (most notably SALSA LipSync).

Requirements

EchoFace's face-tracking component (MediaPipeFaceTracker) is built on top of the MediaPipe Unity Plugin by homuler, which provides the C# bindings for Google MediaPipe's Face Landmarker task inside Unity. This plugin must be installed in the project for EchoFace to compile and function. Otherwise, the Mediapipe, Mediapipe.Tasks.*, and Mediapipe.Unity.Experimental namespaces that MediaPipeFaceTracker depends on are unavailable.

In addition, a Face Landmarker task model asset (a .bytes file) must be assigned in the Inspector (see Face Landmarker Model Asset below). Note that this model must support both facial landmarks and blendshapes to drive the avatar correctly. The MediaPipe Unity Plugin already provides a compatible model out of the box, located directly in its sample/streaming assets, so it only needs to be referenced in the corresponding field.

Architecture

EchoFace is composed of a small number of independent components, each with a single responsibility:

Component	Responsibility
EchoFace	The animation logic. Applies an incoming FaceData frame to the avatar's mesh and bones. Has no knowledge of where the data comes from.
MediaPipeFaceTracker	The tracking source. Wraps MediaPipe's Face Landmarker task against the locally active webcam and raises an event with the latest detection result once per frame.
EchoFaceNetworkBridge	The network layer. Packs tracked frames into a compact binary payload, sends them from the owning client to the server, and broadcasts them from the server to every client.
FaceData / FaceBlendshape / FaceLandmark	Shared data structures: a frame's blendshape weights and the three tracked landmarks (chin, left/right upper eyelid), indexed via enums.
EchoFaceController	The user-facing control layer. Binds a hotkey to enable/disable the facial animation and manages conflicting components.

Data flow

Webcam
  └─▶ MediaPipeFaceTracker                            (owner: detects locally)
        └─▶ EchoFaceNetworkBridge                     (owner: packs + sends)
              └─▶ ServerRpc ─▶ Server
                    └─▶ ClientRpc ─▶ everyone, including the owner
                          └─▶ EchoFaceNetworkBridge   (everyone: unpacks)
                                └─▶ EchoFace          (everyone: animates)

Only the webcam of the owning client is ever read. All clients, including the client that owns it, receive and use the same data. This means that the avatar looks identical to everyone seeing it.

Because the network payload uses unreliable delivery and a monotonically increasing timestamp, out-of-order or dropped packets are simply discarded rather than causing glitches. Only the most recent face pose is ever relevant.

Turning Facial Animation On and Off

Press F10 to toggle EchoFace for your own avatar. This binding is resolved via SEEInput.ToggleEchoFace(), so it follows whatever key SEE's central input system has configured for this action (F10 by default). To remap the key, refer to SEE's input settings.

What happens on toggle:

• Enabling: EchoFace is enabled along with its linked MediaPipeFaceTracker, which starts webcam capture and MediaPipe processing, while any conflicting components on the avatar (see below) are disabled.
• Disabling: EchoFace is disabled, and the tracker is disabled together with it, releasing the webcam. The conflicting components are restored to whatever enabled state they had before. The avatar's face and bones are automatically reset to their initial rest pose.
• A short popup in the corner of the screen confirms the new state ("EchoFace enabled" / "EchoFace disabled").

Whether EchoFace starts enabled or disabled can be configured in the Inspector (see Start Enabled below). By default, it starts disabled.

Note: Animation for facial hair, such as beards or moustaches, is currently not supported. Because they are typically attached as separate child objects with their own independent blendshapes, EchoFace's target CC_Base_Body does not deform them. Consider using avatars without facial hair (e.g., Eddy) until a solution is implemented.

Handling of Conflicting Components

Many avatar prefabs already come with their own facial-animation system, most commonly the SALSA LipSync Suite (lip-sync, eye-blink, and emote components). Running SALSA and EchoFace at the same time causes both to fight over the same bones and blendshapes, producing visibly broken animation.

EchoFaceController resolves this automatically: whenever EchoFace is enabled, it looks through the Behaviours attached to the avatar and disables every one whose class name matches an entry in a configurable list (Conflicting Component Names). By default, this list contains:

• SALSA
• EmoteR
• Silence Analyzer
• Eyes
• Queue Processor
• SalsaDissonanceLink

When EchoFace is disabled again, each of these components is restored to its original enabled state. If a component was already disabled for some other reason before EchoFace ever touched it, it stays disabled.

Note for Developers: If you implement or integrate another module that drives the avatar's face, jaw, or eye movements, add its component class name to Conflicting Component Names in the Inspector to prevent animation conflicts while EchoFace is running.

Inspector Settings

EchoFace

The animation engine, attached to the avatar prefab itself.

Group	Field	Effect
Avatar	Skinned Mesh Renderer	The renderer whose blendshapes are driven. Auto-detected from the avatar skeleton's configured body part if left empty.
Face Animation	Enable Face Animation	Master switch for all blendshape-driven facial animation.
	Enable Viseme Synthesis	Turns on synthesized mouth-shape ("viseme") blendshapes for speech, computed from tracked mouth blendshapes rather than tracked directly.
	Smoothing Rate	Exponential smoothing applied to general blendshapes. Lower values are smoother but slower to react.
	Viseme Smoothing Rate	Separate smoothing rate for viseme blendshapes, typically faster than general smoothing.
	Eye Squint Power	Exponent applied to eye-squint blendshapes to make stronger squints more pronounced. Higher values exaggerate the effect further.
Head Rotation	Enable Head Rotation	Toggles head-bone rotation driven by tracked landmarks.
	Head Transform	The bone to rotate. Auto-detected from the avatar skeleton if left empty. Head rotation is disabled if it cannot be found.
	Rotation Smoothing Rate	Exponential smoothing applied to head rotation.
	Tilt Correction	Manual pitch offset (degrees) to compensate for the webcam's viewing angle. Also offsets vertical eye-look rotation.
Eye Rotation	Enable Eye Rotation	Toggles eye-bone rotation driven by eye-look blendshapes.
	Left/Right Eye Transform	The eye bones to rotate. Auto-detected from the avatar skeleton if left empty.
	Eye Rotation Smoothing Rate	Exponential smoothing applied to eye rotation.
	Eye Look Scale	Scaling factor (degrees) converting eye-look blendshape intensity into an actual rotation angle.

MediaPipeFaceTracker

The tracking source, driving MediaPipe's Face Landmarker against the active webcam.

Group	Field	Effect
Model Configuration	Face Landmarker Model Asset	The .bytes MediaPipe Face Landmarker task model. If missing, initialization fails and an error is logged.

EchoFaceNetworkBridge

The network layer that connects a MediaPipeFaceTracker to an EchoFace across the network.

Group	Field	Effect
Target & Sources	Echo Face	The local EchoFace that received data is applied to. Auto-resolved from the same GameObject if left empty.
	Tracker	The local MediaPipeFaceTracker to listen to on the owning client. Auto-resolved from the same GameObject via GetComponent if left empty.

EchoFaceController

The control layer that binds the hotkey and manages conflicting components.

Group	Field	Effect
Target	Echo Face	The EchoFace to toggle. Auto-resolved from the same GameObject if left empty.
	Face Tracker	The MediaPipeFaceTracker whose enabled state is kept in sync with EchoFace. Auto-resolved via GetComponent if left empty.
Initial State	Start Enabled	Whether EchoFace and its linked tracker should be enabled automatically upon scene start, rather than starting disabled.
Popup UI	Popup Duration	How long, in seconds, the on-screen confirmation popup stays visible.
	Background Color	Background color of the popup panel.
Conflict Resolution	Conflicting Component Names	Class names of components disabled while EchoFace is active and restored afterward.