skin-tokens.cpp

A C++23/GGML port of SkinTokens / TokenRig for automatic skeleton and skin-weight generation on CPU or Vulkan.

Skin weights say how strongly every mesh vertex follows each bone. Without them, moving a skeleton does not deform the character surface correctly. SkinTokens takes a static mesh, predicts a suitable skeleton and its vertex weights, and writes a portable rigged GLB.

Install Git, CMake 3.25 or newer, Ninja, a C++23 compiler, nlohmann-json, and Vulkan headers, loader, and shader compiler. Vulkan can be disabled with -DSKINTOKENS_ENABLE_VULKAN=OFF. From a source checkout:

git submodule update --init --recursive
cmake -S . -B build/release -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build/release -j
cmake --install build/release --prefix ./dist

The ggml/ submodule is pinned to an official upstream commit. During CMake configuration, the project copies it into the build directory and applies the ordered compatibility patches in patches/ggml/; the submodule checkout is never modified. Reconfiguration reuses the prepared copy while the upstream revision and patch hashes are unchanged. Advanced builds using a GGML tree that already contains equivalent fixes can pass -DSKINTOKENS_APPLY_GGML_PATCHES=OFF.

The install contains the shared library, C and C++ headers, CLI, GGML runtime, and enabled dynamic backends. To test from the build tree:

ctest --test-dir build/release --output-on-failure
./build/release/bin/skintokens-cli inspect models/SkinTokens-GGUF/F16

Nix is optional. It supplies the build tools and Vulkan development packages, but does not replace the normal CMake build:

nix develop
cmake --preset debug
cmake --build --preset debug -j
ctest --preset debug

The sanitizer build is the normal development lane:

nix develop
cmake --preset asan-ubsan
cmake --build --preset asan-ubsan -j
ctest --preset asan-ubsan

Downloading the converted F16 GGUF bundle is the recommended setup. With the Hugging Face CLI:

hf download LocalAI-io/SkinTokens-GGUF \
  --include "F16/*" \
  --local-dir models/SkinTokens-GGUF

Use models/SkinTokens-GGUF/F16 as MODEL_DIR in the commands below. The repository also contains an F32 bundle for numerical parity work; normal inference should use F16.

The standalone path needs only a static .glb mesh. It generates the skeleton and learned skin weights, then stores a one-frame rest pose in the output so it opens as a conventional skinned glTF asset:

./build/release/bin/skintokens-cli rig \
  models/SkinTokens-GGUF/F16 \
  character.glb character-rigged.glb \
  --device vulkan

The model accepts arbitrary triangle meshes, although—as with any learned rigging model—results are strongest on shapes resembling its training distribution. --postprocess enables the upstream surface-locality heuristic; omit it to preserve the raw learned weights exactly.

Use skin when the armature already exists. A single rigged GLB can provide both the geometry and static or animated skeleton; its old weights are ignored:

./build/release/bin/skintokens-cli skin \
  models/SkinTokens-GGUF/F16 \
  character-rigged.glb character-rigged.glb character-reweighted.glb \
  --device vulkan --fit none

Alternatively, pass separate mesh and skeleton GLBs. The default --fit global uses a single uniform scale and translation to match vertical extent and centre; it retains every relative joint position and bone-length ratio. Use --fit none when both files already share coordinates.

--fit articulated is an explicitly experimental alternative for recognized humanoid arm chains. A motion-only skeleton offers two length-exact poses: its supplied rest pose, and its own first animation frame. This mode measures both against the mesh surface along the arm bones and keeps whichever actually runs inside the arms, so a T-pose rig driving a character generated with its arms lowered adopts the pose the clip already provides. Doing so also makes the bind pose and frame zero identical, so playback starts without warping the mesh. An arm chain that still misses is then posed by analytic two-bone inverse kinematics toward conservative mesh targets, preserving the globally scaled upper-arm, forearm, and hand lengths exactly. That fallback's invariant follows the template-skeleton embedding objective in Baran and Popovic's Pinocchio, but it is deliberately a small pose stage rather than a copy of Pinocchio's complete LGPL rigging and weight-generation library. A fuller distance-field embedding implementation can replace this isolated stage later.

Safe checkpoint extraction, checked GGUF loading, CPU/Vulkan backend selection, the complete TokenRig grammar and Qwen3 stack, Kimodo animation import, and skinned GLB export are implemented. The F32 Qwen stack reaches 8.1e-6 relative L2 on CPU; Vulkan reaches 7.1e-4 for the final hidden state and 1.4e-3 for logits.

The learned binding path runs the released Michelangelo mesh encoder, Qwen TokenRig policy, FSQ code expansion, condition encoder, and chunked SkinVAE decoder through GGML. It samples the same 54,000 surface points as upstream and caps dense decoder queries at 16K vertices per graph. The skin command uses a supplied static or animated skeleton. The rig command enables upstream's unconstrained skeleton generation, which remains experimental because the released policy can produce unsuitable topologies. --geometric is an explicit non-learned diagnostic.

Upstream's existing-skeleton interface is deliberately generic: --use_skeleton retains an armature found in the input and generates skin only. It does not promise that every arbitrary hierarchy is in-distribution or prescribe a retargeting method. Imported GLBs are tokenized as generic articulations. The released checkpoint stores order_config, while TokenizerPart.parse reads order, so its executable imported-GLB path emits no spring/part token and does not activate a hidden Mixamo-specific inference mode from joint names.

The full 54K-point Michelangelo encoder matches the corrected F32 PyTorch reference at 6.5e-4 relative L2 on Vulkan. A constrained 120-step fixture checks the production Qwen prefix and greedy policy decision at every step; Vulkan reproduces every reference code. The full 54K-point SkinVAE and final raw learned binding are checked independently. CPU reaches 3.6e-6 decoder relative L2, identical top-four bones on every vertex, and 1.4e-7 animated-vertex relative L2. The representative Vulkan fixture reaches 9.1e-4 decoder relative L2, with 99.96% of final top-four slots shared and 5.2e-4 animated-vertex relative L2. The optional voxel_skin heuristic is available with --postprocess, matching the upstream demo's opt-in setting rather than being silently enabled.

Use --device vulkan, --device cpu, or --device auto at runtime.

Set SKINTOKENS_PROFILE=1 to print opt-in wall-clock timings for preprocessing, the mesh and SkinVAE encoders, TokenRig generation, binding integration, and model loading. TokenRig's summary also reports graph count, host/device transfer volume, transfer time, synchronized graph-compute time, and device-local KV cache cloning:

SKINTOKENS_PROFILE=1 ./build/release/bin/skintokens-cli skin \
  models/SkinTokens-GGUF/F16 mesh.glb skeleton.glb result.glb --device vulkan

TokenRig keeps an F32 K/V cache on the selected backend. Surviving beams retain their cache slots, while additional children of the same parent are cloned device-to-device. Cache memory therefore grows with context length and the configured beam count; lower --beams or --max-tokens when device memory is limited.

This is an advanced reproducibility path; most users should download the GGUF bundle above. The two official checkpoints are MIT-labelled but use Lightning pickle containers. Downloads are explicit, hash-checked, and excluded from git:

./scripts/download_models.sh checkpoints
docker build -t skintokens-reference:2.7 reference
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/work" \
  skintokens-reference:2.7 \
  --tokenrig /work/checkpoints/grpo_1400.ckpt \
  --skin-vae /work/checkpoints/last.ckpt \
  --output /work/generated/safe
python scripts/convert_to_gguf.py \
  --tokenrig generated/safe/tokenrig.safetensors \
  --skin-vae generated/safe/skin-vae.safetensors \
  --config generated/safe/config.json \
  --output models/skintokens-f32 --ftype 0

The normal converter accepts safetensors only. Every GGUF component records the upstream revision and both intermediate SHA-256 identities, and model load rejects mixed bundles.

Trellis2 can generate and remesh a humanoid, Kimodo can generate its motion, and skin-tokens.cpp can bind that motion hierarchy to the mesh. This is a concrete full animated-asset pipeline, not a dependency or requirement of SkinTokens:

trellis2cpp/build/examples/mesh2glb \
  trellis-mesh.t2mesh trellis-remeshed.glb 2048 \
  --print 0.5 0.0166667 --quad 20000
./build/release/bin/skintokens-cli skin \
  models/SkinTokens-GGUF/F16 \
  trellis-remeshed.glb kimodo-animation.glb character-animated.glb \
  --device vulkan

When the supplied skeleton is detected as SOMA30, it can optionally be retargeted to the 52-joint order published in upstream's configs/skeleton/mixamo.yaml using an explicit semantic motion map:

./build/release/bin/skintokens-cli skin \
  models/SkinTokens-GGUF/F16 \
  character.glb kimodo-soma30.glb character-mixamo.glb \
  --device vulkan --retarget-soma-to-mixamo52

./build/release/bin/skintokens-cli retarget-check kimodo-soma30.glb

retarget-check rotates each mapped source joint by +/-30 degrees around all three axes and compares corresponding FK trajectories in normalized body space. It also reports full-clip joint-position and foot-velocity errors. This tests retargeting independently of learned skin weights. Retargeting is off by default and appears in the demo only after strong SOMA30 detection.

For an unconstrained SkinTokens result, retain its mesh-fitted generated rig and learned weights and transfer SOMA30 motion onto that hierarchy afterwards:

./build/release/bin/skintokens-cli rig \
  models/SkinTokens-GGUF/F16 character.glb character-native.glb --device vulkan
./build/release/bin/skintokens-cli retarget-generated \
  character-native.glb kimodo-soma30.glb character-animated.glb

./build/release/bin/skintokens-cli retarget-generated-check \
  kimodo-soma30.glb character-animated.glb

This structural recognizer maps the 22 body joints shared with SOMA30 and tolerates different generated hand subtrees. Unmatched fingers inherit the hand motion in a neutral local pose by default; --map-fingers copies SOMA's two endpoint tracks across those branches. Low-confidence or non-humanoid generated skeletons are rejected rather than matched by spatial proximity. retarget-generated-check compares rest-relative joint displacement in body- height units and reports the mean, maximum, worst frame, worst joint, and per-joint maxima.

Both .glb and trellis2cpp's versioned .t2mesh are accepted by the CLI as mesh input. Skinning should use the manifold-wrapped, quad-remeshed Trellis export, not its dense raw marching-cubes reconstruction: the latter is millions of triangles with generation topology that is unsuitable for animation. Direct quad remeshing is not sufficient for raw reconstruction topology; Alpha Wrap must clean it first. GLB scene-node transforms and vertex colours are preserved; atlas textures are not yet round-tripped. The Kimodo rig is fitted to the mesh before conditioning, while its animation and hierarchy are retained in the output. SkinVAE predictions are decoded on the upstream 54K cloud and transferred to source vertices with the same SciPy cKDTree inverse-distance 8-neighbour interpolation, then selects and renormalizes the greatest four influences for glTF. With --postprocess, the optional upstream heuristic first multiplies those channels by its voxel_skin surface-geodesic matrix. The C++ diagnostic retains Open3D voxel ordering and SciPy nearest-neighbour tie behaviour. Generation currently recomputes the Qwen graph without a KV cache and may take several minutes.

skintokens.h is a flat C11-compatible API. It uses an opaque model handle, caller-owned fixed error buffers, explicit defaults, and status returns; C++ exceptions never cross the ABI. Returned backend and last_error strings are borrowed from the model and remain valid until its next call or destruction.

#include <skintokens/skintokens.h>

char error[512];
st_model *model = NULL;
st_runtime_options runtime = st_default_runtime_options();
runtime.device = ST_DEVICE_VULKAN;
if (st_model_load("models/SkinTokens-GGUF/F16", &runtime, &model,
                  error, sizeof(error)) != ST_OK) {
    /* report error */
}

st_generation_options generation = st_default_generation_options();
generation.surface_postprocess = 1;
int learned = 0;
st_status result = st_skin_files(model, "character.glb", "motion.glb",
                                 "animated.glb", ST_FIT_GLOBAL_SIMILARITY,
                                 &generation, &learned,
                                 error, sizeof(error));
st_model_free(model);

The generated-rig retarget surface uses opaque option and match handles too, so PureGo and other FFI callers never have to predict a C structure layout:

st_retarget_options *options = NULL;
st_humanoid_match *match = NULL;
st_retarget_options_create(&options, error, sizeof(error));
st_retarget_options_set_minimum_confidence(options, 0.85f,
                                           error, sizeof(error));
st_humanoid_match_glb_file("character-native.glb", options, &match,
                           error, sizeof(error));
st_retarget_soma30_glb_files("character-native.glb", "motion.glb",
                             "character-animated.glb", match, options,
                             error, sizeof(error));
st_humanoid_match_free(match);
st_retarget_options_free(options);

st_rig_file generates a skeleton and weights; st_skin_files generates weights for a supplied static or animated skeleton. Pass the same GLB as both mesh and skeleton paths for a single rigged asset. st_inspect_glb_file reports mesh, skin, skeleton, animation, and recognized-rig metadata without loading GGUF weights. The public parsing and error-buffer surface is exercised by the ASan/UBSan libFuzzer build; GGUF loading is intentionally kept outside that fuzzer.

The dependency-free Go/WebGL demo is upload-first. Choose Skeleton + skin for an unrigged mesh, or Skin weights only for either one rigged GLB or separate mesh and skeleton GLBs. Kimodo animation GLBs work directly as the separate skeleton input. Skeleton + skin also accepts an optional Kimodo SOMA30 animation: the demo keeps the generated native rig, transfers the motion onto it, and offers both files for download. Playback appears only when the supplied hierarchy is animated. Separate inputs default to the length-preserving global fit and offer articulated fitting as an experimental checkbox. SOMA30 retargeting appears only when that rig is detected. The demo keeps persistent history, runs one bounded worker, visualizes joint influences, and exports reusable GLBs. For a generated rig driven by SOMA30, the viewer can overlay the normalized source motion in blue, draw the retarget error vectors, and colour mapped joints by divergence. Upload selections are retained when switching workflows: an unrigged Skeleton + skin mesh becomes the separate character mesh in Skin weights only, while a rigged input becomes the one-file asset. The optional driving motion has an explicit Clear button.

cd demo
go run . \
  --listen 127.0.0.1:8095 \
  --cli ../build/release/bin/skintokens-cli \
  --model ../models/SkinTokens-GGUF/F16

Open http://localhost:8095. Paths and the listen address are flags; no local machine paths are compiled into the C++ library or web server. Optional giraffe-example paths default to disabled.

Capture operation boundaries from the safe checkpoint in the pinned reference image, then compare every Qwen, mesh-encoder, SkinVAE, and generation boundary. The unconstrained fixture additionally records the skeleton/skin switch and upstream's unusual use of model EOS as the final FSQ code:

docker build -t skintokens-reference:2.7-parity reference
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/work" \
  --entrypoint python skintokens-reference:2.7-parity \
  /work/reference/capture_qwen_layer.py \
  --weights /work/generated/safe/tokenrig.safetensors \
  --output /work/fixtures/reference-f32 --device cpu --dtype float32

./build/release/bin/skintokens-qwen-parity \
  models/skintokens-f32 fixtures/reference-f32 cpu
./build/release/bin/skintokens-mesh-parity \
  models/skintokens-f32 fixtures/reference-f32 cpu
./build/release/bin/skintokens-vae-parity \
  models/skintokens-f32 fixtures/reference-f32 cpu
./build/release/bin/skintokens-generation-parity \
  models/skintokens-f32 fixtures/reference-f32 cpu
./build/release/bin/skintokens-qwen-batch-parity \
  models/skintokens-f32 fixtures/reference-f32 cpu

The batch checker compares two different token sequences evaluated together against two serial evaluations, then repeats the comparison after a shared KV prefill and a cloned two-child beam branch. This guards batching, cache strides, and beam-slot cloning on both CPU and Vulkan; configure SKINTOKENS_QWEN_FIXTURE_DIR to include it in the opt-in CTest model suite.

Final binding acceptance captures normalized top-four weights and reference deformed vertices. The full checker runs the production GGML decoder and skinning evaluator in one process:

./build/release/bin/skintokens-full-binding-parity \
  models/skintokens-f32 fixtures/animation-parity cpu
./build/release/bin/skintokens-full-binding-parity \
  models/skintokens-f32 fixtures/animation-parity vulkan

For the strict Vulkan diagnostic, disable reduced-precision matrix paths:

GGML_VK_DISABLE_F16=1 GGML_VK_DISABLE_COOPMAT=1 \
GGML_VK_DISABLE_COOPMAT2=1 GGML_VK_DISABLE_GRAPH_OPTIMIZE=1 \
  ./build/release/bin/skintokens-qwen-parity \
  models/skintokens-f32 fixtures/reference-f32 vulkan

SkinTokens is a learned skin-weight representation. TokenRig is the complete mesh-to-rig system: a Michelangelo point encoder, Qwen3-0.6B causal model, and conditional FSQ-VAE decoder. This use is therefore within its intended scope, not an attempt to treat SkinTokens as a motion generator.

The upstream repository and Hugging Face model card identify code and weights as MIT. This project is Apache-2.0 and keeps upstream attribution in NOTICE.