GitHub - ryoyaks/SyncRig: Open-source motion capture & 3D pose reference for creators. Drop in a photo, video, or webcam — get an editable 3D pose. Runs locally, exports to Blender, broadcasts via VMC.

Open-source motion capture & 3D pose reference, made for creators.

What SyncRig does

Capture motion from a photo or video

Drag any photo or video into SyncRig and you get an editable 3D pose in seconds. Inspect unfamiliar poses from any angle, skip the photo-hunting and figure-posing busywork, and put your time back into the actual creative work.

Multiple input sources

Photos, video files, and live webcam — same tool whether you want a single pose reference, a recorded motion, or a VTuber stream.

Not happy with the result? Touch it up

Every capture can be corrected:

Face — integrates the VTube Studio iPhone Face ID channel over the VMC protocol, so you can bring iPhone facial capture into SyncRig (or broadcast back to your VTuber app).
Hands — MediaPipe fills in wrist position and per-finger joint rotations, useful when the primary model misses subtle hand detail.
Any bone — Edit mode lets you select any humanoid bone, rotate it with an FK gizmo, and press K to commit a keyframe on the timeline. The original mocap is preserved; edits live in a sparse overlay (<srrec>.edits.json).

Where it fits

SyncRig in the broader “turn a reference into 3D pose data” space:

	Manual 3D pose tools	AI 2D pose detectors	Cloud mocap services	SyncRig
Pose source	manual joint dragging	photo (flat 2D)	webcam / video	photo or video
Output	editable 3D	flat 2D heat-map	recorded mocap	editable 3D
Where it runs	usually local	usually local	mostly cloud	100 % local
Pricing	mostly paid / subscription	free	subscription	free / open source
Swappable model	✗	✗	✗	✓ (provider plug-in API)

The primary audience is illustrators and concept artists who want 3D reference without spending fifteen minutes posing every limb. Animators get the same engine plus a Blender add-on, BVH/glTF/FBX export, and a timeline that lets you stack keyframe edits on top of the mocap track.

Creative-software integrations

To slot mocap into the workflow you already have, we ship downstream integration plugins:

Blender add-on (launched) — live mocap into the active armature, plus multi-format export and keyframe editing.
VMC Out bridge (launched) — broadcast the rig to VTube Studio / Warudo / EVMC4U as a Performer for downstream pipelines.
Unity / Unreal plug-ins (in progress) — let game and real-time engine developers consume SyncRig mocap data directly inside the engine.

Pluggable model API

SyncRig is built around a provider plug-in architecture: every pose / mesh model is an independent provider that you can swap inside the app. Alongside the bundled MediaPipe / ROMP / SAM 3D Body, we publish a Provider Template so you can wire up new SOTA models — or your own fine-tunes — without touching the rest of the pipeline. Everything downstream (timeline, edits, export) just works.

Quick start (end users · Windows installer)

Fastest path:

Grab the latest installer from Releases:
- SyncRig-Setup-cuda-*.exe — for NVIDIA GPUs. Faster, all providers available.
- SyncRig-Setup-cpu-*.exe — laptop / no discrete GPU. Smaller, still ships MediaPipe.
Run it, pick an install path, launch SyncRig (CUDA) / SyncRig (CPU).
Drop a photo or short video into the Live or Process tab.

MediaPipe is ready out of the box. Other providers (SAM 3D Body, ROMP, SMPL) install one-click from the in-app Extensions tab; weights download in the background.

Tutorials

Five common workflows — pick whichever matches what you want to do.

1. Live mocap from a webcam (Live mode)

The fastest way to try things — good for playing around, VTuber streaming, or feeding Blender directly.

Switch to the Live tab and pick an input — webcam by default, or you can choose screen capture or a local video file.
Pick a provider from the right-hand dropdown (default is MediaPipe Holistic — full body + hands + face, runs on CPU).
Hit Start. The 3D viewport mirrors your motion in real time.
Want to record it? Hit the ● Record button at the top — once to start, once to stop. It auto-saves as .srrec, which you can later scrub in the Process / replay tab.
Want to feed VTube Studio / Warudo? Open Settings → VMC Out and enable broadcast; the downstream app picks the rig up.

Live mode can also take a video file, but it uses a causal (real-time) filter — if the model can't keep up, frames drop and the output is jitterier. For clean results, use the Process tab instead (next section). Live-on-video is more for quick previews, trying out a provider, or broadcasting while you play.

2. Pose reference from a photo

Switch to the Process tab.
Drag the photo in (or click Select file).
SAM 3D Body is the default — it handles stylised input (cosplay, fashion photography) much better than classic pose detectors. You can also switch to ROMP or MediaPipe in the provider dropdown.
After processing, orbit / pan / dolly the 3D viewport to find the angle your composition needs — screenshot it or keep editing.

3. Motion from a video file

Switch to Process, drop the video in.
Wait for the frame-by-frame pass to finish — frame-perfect mode runs sequence-wide forward+backward 1€ filtering for a cleaner track than live.
Scrub the timeline to pick the frame you want — or hit Export to save the whole motion as .srrec / BVH / glTF / FBX.

4. Hand / face touch-ups

When the primary model misses detail:

Open Hands Adjustment or Face Adjustment in the right panel.
Pick a source — hands currently use MediaPipe; face supports iPhone Face ID via VTube Studio.
The adjustment overlays onto the current rig in real time. Press K to commit a keyframe on the timeline.

5. Export to Blender

Inside SyncRig, export any .srrec to BVH / glTF / FBX.
Or install the SyncRig Blender add-on (scripts/build_blender_addon.py produces the zip) — Blender connects to the engine and mirrors mocap onto the active armature live.
Keep animating in Blender — keyframe edits on top of the mocap track live in a sidecar, so the raw take is preserved.

Showcase

This section will collect work made with SyncRig — illustration reference, Blender animations, VTuber streams, experimental pipelines, anything goes.

If you've built something with SyncRig, drop it in Discussions or open a PR to add it here — we'll curate.

Roadmap

Directions we're evaluating / planning to add:

Subject selection in multi-person footage — when the scene has more than one person, let the user pick which one to track instead of leaving it up to the model.
Multi-view input pipeline — feed several camera angles simultaneously and fuse across views for tracks that are sturdier and closer to true 3D than any single view.

Got something you'd like to see? Open an Issue or drop into Discussions.

Developer quick start — from source

git clone https://github.com/ryoyaks/SyncRig.git
cd SyncRig

# Lightweight baseline (MediaPipe only):
uv sync

# Or with all bundled providers:
uv sync --extra smpl --extra sam3d

cd packages/syncrig-web && npm install && npm run build && cd ../..

uv run syncrig-engine serve --serve-web
# → Web UI:    http://127.0.0.1:8765/
# → WebSocket: ws://127.0.0.1:8765/syncrig

For the Blender add-on:

uv run python scripts/build_blender_addon.py
# → dist/syncrig-blender-v*.zip

Blender → Edit → Preferences → Add-ons → Install from Disk → pick the zip → open the SyncRig N-panel → Connect.

Advanced features

Some of these only appear with Settings → Interface → Developer mode enabled.

Live multi-client streaming — one engine, many consumers. Web, Blender, and future Unity / Unreal targets all read the same SyncRigPayload over WebSocket.
Frame-perfect offline mode — process a whole video with sequence-wide forward+backward 1€ filtering for cleaner tracks than the live causal filter.
Record + replay — .srrec captures the full mocap payload (skeleton / mesh / hands / face) plus the original source media; replay rewinds, scrubs, loops, and re-broadcasts.
VRM keyframe editor (Edit mode) — Blender-style FK gizmo, sparse overlay on top of the mocap track, multi-track timeline.
Pose Library — save / apply named rig poses; commits via the same Edit-mode keyframe path.
Multi-format export — .srrec, archived source video, .edits.json, plus retargeted motion as BVH / glTF / FBX or motion JSON.
Engine-side retarget — single source of truth: web and Blender consume the same vrm_pose dict so the avatar can't drift between clients.
Tracking mask — per-modality (body / hands / face / SMPL) toggles applied uniformly across providers.
Click-to-install providers — the Extensions page runs uv sync --extra <name> server-side and streams progress; no terminal needed.
Engine-log mirror — server warnings & tracebacks stream into the web Onscreen Console so a misbehaving provider surfaces where you're looking.
Multi-language UI — English, 日本語, 繁體中文; switch from Settings → Interface.

Providers (bundled)

Provider	Topology	Mesh	Hands / Face	Extra	Notes
MediaPipe Holistic	`mediapipe_33`	—	MP Hand 21 + Face 478 + ARKit blendshapes	default	CPU; fast baseline
ROMP	`smpl_24`	SMPL (6890 v)	—	`--extra smpl`	Multi-person SMPL parametric body
SAM 3D Body	`sam3d_body` (70 MHR)	MHR (~9 k v)	—	`--extra sam3d`	Meta's photo-friendly model; handles stylised input
Hybrid	—	inherits	overlay	—	Pair a body provider with MediaPipe hands / face

Each provider's Extensions card shows a commercial-suitability chip (safe / conditional / non-commercial) sourced from capabilities() so you can see the license footprint without reading docs.

External provider packages

Niche or research-grade providers ship as separate pip packages so the main repo stays focused:

Package	Adds	License
`ryoyaks/SyncRig-model-SMPLerX`	`smplerx` — whole-body SMPL-X (body + 30 finger joints + face) with mesh	S-Lab + SMPL-X (MPI) — non-commercial
`ryoyaks/SyncRig-model-VITPose`	`vitpose` — transformer-based 2D body pose · WIP	Apache 2.0

End users — install from the Extensions tab

No terminal required:

Open the Extensions tab in the app.
Find the Install from Git card and paste the GitHub URL (e.g. https://github.com/ryoyaks/SyncRig-model-SMPLerX).
The first install shows a trust confirmation — the engine then git clones and runs uv pip install for you, streaming progress into the page.
Once installed, the provider shows up in the Live / Process tab dropdown, ready to switch to.

The same flow works for any third-party repo that follows the Provider Template — including your own forks or fine-tunes.

Developers — install from source

uv pip install 'syncrig-model-smplerx[runtime]'

The engine's ProviderRegistry.autoload_entry_points("syncrig.providers") hook discovers the package at next launch; the Extensions page card appears with description + license chip.

Architecture

SyncRig/
├── packages/
│   ├── syncrig-core/      [Python] SDK contract — schema, topology, ABCs, state controllers
│   ├── syncrig-engine/    [Python] Runtime     — input, providers, pipeline, WS server
│   ├── syncrig-blender/   [Python] Blender add-on client
│   └── syncrig-web/       [TS]     React + R3F frontend
├── scripts/
│   ├── export_schema.py             # Pydantic → JSON Schema → TS types
│   ├── verify_mediapipe_retarget.py # Retarget regression harness
│   └── build_blender_addon.py       # Bundle the Blender plug-in
└── tools/                           # srrec → BVH / glTF / motion converters

Data flow

Webcam / Video / Image
   │
   ▼
[Input Source]            syncrig-engine/input/
   │
   ▼
[Provider]                syncrig-engine/providers/
   │ ProviderOutput (skeleton + smpl + mesh + hands + face)
   ▼
[Pipeline Worker]         syncrig-engine/pipeline/worker.py
   │ TrackingMask → 1€ smoothing → engine-side retarget → SyncRigPayload
   ▼
[Recorder, optional]      .srrec
   │
   ▼
[WS Server]
   │
   ├──► Blender add-on  → LatestFrameBuffer → bpy.app.timers → drivers
   └──► Web app         → frame_buffer.ts   → useFrame       → Three.js mutate

Adding a provider

Two paths:

In-tree — drop a myprovider_provider.py under packages/syncrig-engine/src/syncrig_engine/providers/ with a class decorated @ProviderRegistry.register and a capabilities() classmethod. Autoload picks it up on next engine start.
External pip package — declare
```
[project.entry-points."syncrig.providers"]
myprovider = "my_pkg.provider:MyProvider"
```
autoload_entry_points("syncrig.providers") discovers it at engine startup; the package's import-time code registers topology + retarget handler, and the provider appears in the Extensions page.

Both paths share the same Provider ABC + ProviderCapabilities contract.

Documentation

models/README.md — model weights layout, mirror semantics, per-provider install
packages/syncrig-blender/README.md — Blender add-on usage
packages/syncrig-web/README.md — web client architecture

License

The SyncRig code in this repo is Apache 2.0. The LICENSE file covers everything authored here — schema, engine, providers, retarget, web client, Blender add-on.

Third-party assets you pull in at runtime have their own terms. Nothing below is shipped in the repo; this table tells you what license attaches to each thing after you install / download it.

Third-party notices

Component	License	When triggered	Commercial use
MediaPipe (Google)	Apache 2.0	bundled in `mediapipe` PyPI wheel	✅ Unrestricted
torchvision + COCO FasterRCNN	BSD-3	shared person detector	✅ Unrestricted
PyTorch	BSD-3	runtime for every ML provider	✅ Unrestricted
simple-romp code	Apache 2.0	`uv sync --extra smpl`	✅ Unrestricted
ROMP.pkl checkpoint	Apache 2.0	auto-download on first ROMP run	✅ Unrestricted
SMPL parametric model	SMPL Model License (MPI)	user fetches from `smpl.is.tue.mpg.de` (account)	❌ Research only. Commercial use needs Meshcapade license.
SAM 3D Body weights	Meta SAM License	from `ryoyaks/SyncRig-weights` Releases or HF gated repo	⚠ Allowed under § 2 (no military, MAU < 700 M)
SAM 3D Body code (our port)	Apache 2.0 (port) + Meta SAM License (original)	`uv sync --extra sam3d`	Same as Meta SAM License
ViTPose + `transformers`	Apache 2.0	`pip install syncrig-model-vitpose[runtime]`	✅ Unrestricted
VRM 1.0 / three-vrm	Apache 2.0	web client	✅ Unrestricted

Each provider declares its posture via a commercial: "safe" | "conditional" | "non-commercial" chip surfaced in the Extensions page — install with eyes open.

Protocol references

The VMC Out bridge and iPhone Face fusion are implemented from public protocol documentation only. No copyleft (GPL/AGPL) code is referenced or vendored.

VMC Out — Performer → Marionette (OSC over UDP)

VMC Protocol spec — MIT. Wire format implemented from scratch in packages/syncrig-engine/src/syncrig_engine/bridges/vmc_out.py.
python-osc — Unlicense.
gpsnmeajp/EasyVirtualMotionCaptureForUnity (EVMC4U) — MIT. C#/Unity reference for Marionette behaviour; cross-referenced for bone-name conventions (no code vendored).

iPhone Face — VTube Studio iOS "3rd Party PC Clients" (JSON / UDP)

VTube Studio (DenchiSoft) — the source iOS app whose protocol we receive.
DenchiSoft/VTubeStudioBlendshapeUDPReceiverTest — MIT. C# sample receiver; face_inputs/iphone_vts.py is an original Python port written from the spec + observed wire packets.

Acknowledgements

SyncRig stands on top of a stack of excellent open-source work. The project would not exist without these:

Pose / mesh models

MediaPipe (Google) — pose, hand, face landmark models. The baseline that ships out of the box.
SAM 3D Body (Meta) — single-image whole-body mesh + 70-joint MHR skeleton. The default for stills.
ROMP (Yu Sun et al.) — multi-person SMPL parametric body from a single image.
SMPL / SMPL-X (Max Planck Institute for Intelligent Systems) — the parametric body model behind ROMP / SMPLer-X.
SMPLer-X (S-Lab) — whole-body SMPL-X regression; provided through the external SyncRig-model-SMPLerX package.
ViTPose (ViTAE) — transformer-based 2D body pose; provided through SyncRig-model-VITPose.
DINOv3 (Meta) — vision backbone used by SAM 3D Body.

3D, rendering, rigging

Three.js (mrdoob & contributors) — the WebGL renderer behind the web viewport.
@react-three/fiber + drei (Poimandres) — declarative Three.js bindings used throughout the web client.
three-vrm (@pixiv) — VRM 1.0 loading + humanoid rig.
VRM Consortium — the VRM 1.0 spec that defines the avatar contract every consumer in SyncRig agrees on.

Tooling & runtime

Blender (Blender Foundation) — the headline DCC target for the add-on.
PyTorch (Meta + community) — runtime for every ML provider.
OpenCV — video / image I/O, image preprocessing.
React (Meta + community) — UI library.
Electron + electron-builder — desktop wrap + the custom dark-themed installer in SyncRig-electron.
uv (Astral) — Python toolchain that powers the click-to-install Extensions flow.
Pydantic — payload schema validation; single source of truth for the wire format.
aiohttp — engine HTTP / WebSocket server.
msgpack-python — wire encoding for SyncRigPayload.

Interop / protocol

VMC Protocol — Performer/Marionette spec used by SyncRig's VMC Out bridge.
python-osc — OSC encoding for VMC Out.
EVMC4U — Unity reference for VMC Marionette bone conventions.
VTube Studio (DenchiSoft) — iPhone Face source protocol; sample receiver under VTubeStudioBlendshapeUDPReceiverTest.

Inspiration

Kalidokit (yeemachine) — referenced for the per-bone rest-to-live alignment pattern that powers SyncRig's hand thumb retarget.
Magic Poser / Design Doll — validated the demand for editable 3D pose reference among illustrators. SyncRig occupies the same product slot — free, open, local, and auto-generated from a photo or video rather than manual joint dragging.
ShibaPaw — the MediaPipe-to-Blender project that informed our retarget logic and the Blender add-on's skeleton sync.

If a project's work is in SyncRig and we've missed it here, please open an issue — we'd like to credit you properly.

Name		Name	Last commit message	Last commit date
Latest commit History 228 Commits
.github/workflows		.github/workflows
asset		asset
docs		docs
models		models
packages		packages
scripts		scripts
sessions		sessions
tools		tools
.gitignore		.gitignore
.gitmodules		.gitmodules
.python-version		.python-version
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.ja.md		README.ja.md
README.md		README.md
README.zh-TW.md		README.zh-TW.md
VERSION		VERSION
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Folders and files

Latest commit

History

Repository files navigation

What SyncRig does

Capture motion from a photo or video

Multiple input sources

Not happy with the result? Touch it up

Where it fits

Creative-software integrations

Pluggable model API

Quick start (end users · Windows installer)

Tutorials

1. Live mocap from a webcam (Live mode)

2. Pose reference from a photo

3. Motion from a video file

4. Hand / face touch-ups

5. Export to Blender

Showcase

Roadmap

Developer quick start — from source

Advanced features

Providers (bundled)

External provider packages

Architecture

Data flow

Adding a provider

Documentation

License

Third-party notices

Protocol references

Acknowledgements

Pose / mesh models

3D, rendering, rigging

Tooling & runtime

Interop / protocol

Inspiration

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages