Reze MiPo 5.0

Previously MiKaPo. Same project, now named for the family it belongs to — alongside reze-engine, reze-design, reze-studio and reze-rig.

The solver was rebuilt this release, one technique at a time, each webcam-tested before the next went in — and then measured against recorded video by a harness built for the purpose.

Capture

• The z channel is filtered alone. x and y come off the image; z is inferred and carries the noise. Low-passing that one channel before solving took torso wobble from 6.7°/frame to 0.3°.
• Every signal gets the filter it deserves. A spine turns slowly and a fingertip does not, so each measured direction is smoothed as its own named signal — in the frame it was measured in, renormalised after, with cutoffs that follow the body and slow for a subject small in frame. About a quarter less movement per frame across the arms on noisy footage.
• Roll is solved from a witness, and guarded. The forearm reveals how the upper arm is twisted, weighted by how observable that twist is. A witness basis flips when the limb it reads passes through the parent's own axis, so past 110° of disagreement it fades out: that is the basis flipping, not a limb rolling half a turn between frames.
• A body faces one way. A neck may turn a hundred degrees from its chest and a chest sixty from its hips, and what is asked past that is eased back until it fits. The trunk, the chest and the head are each solved from different landmarks and nothing used to require them to agree, so a detector reading one of them backwards produced a head facing behind a chest facing front.
• The spine splits the twist. What the chest does relative to the hips is halved between 上半身 and 上半身2, so the waist carries 17° where it used to carry all 34°. Which square root of that twist is taken is chosen for continuity — the nearer one changes as a body turns past a half circle, and taking it blindly threw the neck, head and both arms a half turn while the chest still composed correctly.
• The arm's twist goes on the bone built to carry it. 腕捩 and its three followers exist so a rotating forearm carries the sleeve around instead of pinching. Up to 96° of that turn had been sitting on 腕 and twisting the whole upper arm from the shoulder.
• Limbs the camera cannot see stay still. A bone is only as sound as the landmark its segment is least sure of, so the gate takes the worst rather than the average — a hip at 0.62 no longer carries a knee at 0.23 past it. Legs hold a slow state on top: out of shot half a second and they rest. Framed from the waist up, leg movement per frame falls from about 12° to about 3°.
• A frame whose bones are the wrong length is refused. The detector's confidence does not report this — it rates a body reconstructed inside out at 0.999 — but the distance across the shoulders is a bone, and when a detection collapses it drops to a quarter of what it was.
• Dropouts crossfade, and a gap is not a loss. A bone holds its last measurement for 250ms, eases to rest over 500ms, and eases back over 250ms. Ordinary detection gaps cost nothing.
• Capture is in place. センター carries height, so crouches and level changes land while the character holds its mark.
• The display plays instead of chasing. A cursor advances along the media timeline and interpolates the two results bracketing it: ~10ms less latency and 7 points more amplitude on fast motion.

Export

An export has every frame's neighbours, which live capture never will.

• Landmarks are cleaned before anything is solved — a median of three removes single-frame outliers, then a zero-phase fit smooths what remains, hardest on z.
• The take is solved once, forwards, with the filters opened up because a zero-phase fit follows and removes the lag they leave: 3.14° → 1.73° RMS against ground truth.
• Keyframes are reduced to the ones that decide a track's shape, bounded at half a degree and three millimetres — about a seventh of the keys, for a file an animator can open and edit.
• Motions are named after the footage they came from.

Interface

• The capture panel is a window: drag it, resize it, and it comes back where you left it.
• One media source. The file's own type decides whether it opens as video or as a still.
• Uploads survive a refresh — your model and your footage are kept in IndexedDB.
• The design language is now shared with reze.design and reze.studio, down to the wordmark.

Measuring it

harness/ records what MediaPipe actually says about a real clip once, in a headless browser, then replays that recording through the real solver in plain Node — reporting per bone how far it moves between frames, its worst single step, and its roll. Roll is the channel a landmark preview can never show: the stick figure is identical whichever way a limb is twisted, which is why several of the bugs above survived every earlier round of testing and were caught in an afternoon once something was watching it.

Try it

v4.0.0 — motion worth keeping

Up to 3.x this was proof that MMD mocap could run in a browser at all: a pose followed a person, live, and the file it wrote was rotations. 4.0 is about the capture being right — the body stands on the ground, joints bend the way joints bend, the model's own geometry keeps limbs out of itself, and a bad frame of detection is refused rather than displayed.

The body is placed, not just posed
センター carries a height and the leg IK bones carry positions, so crouches, level changes and weight drops survive into the file. Placement is exact rather than inferred: the legs are walked forward from the hips and the body is dropped until its lowest part rests where it rests in the model's bind pose. Feet, or hips when the feet are not under you — which is what stops a floor roll from bouncing the model. Exports keyframe 足ＩＫ and switch leg IK on, so a motion is a native MMD leg rig, editable the way MMD users expect.

Two honest limits: jumps do not register (both feet airborne is indistinguishable from standing in hip-centred landmarks), and once the torso leaves vertical the height is held rather than guessed.

Anatomy the solver now respects

• The clavicle takes its share of arm elevation, roughly 2:1 with the humerus. 腕 hung straight off 上半身 before, so the arm performed every raise alone — impossible past ~30°, and it looked it.
• 穿模 is answered with the model's own rigid bodies. An MMD model already carries its author's approximation of the character as capsules; MMD never tests them against each other, which is exactly how a hand ends up inside a chest.
• Finger bend is measured about its own axis instead of borrowing the total rotation angle, so splay no longer folds knuckles backwards.

Signal quality

• Hands are gated on real confidence. MediaPipe's hand landmarks carry no visibility field, and a lost hand collapses toward the origin rather than vanishing — still 21 valid-looking points.
• Smoothing reads true angular velocity, and refuses accelerations no limb could make. A speed-adaptive filter is otherwise defenceless against a one-frame outlier, which looks exactly like a fast move.
• Exports smooth in both directions: a Savitzky-Golay pass over the finished take — 59% less jitter, 99% of peak amplitude kept, zero phase shift.

Fixes

• Stills are stills: the landmarker resets between images, and the pose applies unfiltered and untweened.
• Video duration is resolved before use; Infinity from WebM would have made the conversion loop run forever.
• Space toggles playback; buttons release focus after a mouse click.
• The tuning panel is gone. The filter constants are the solver's business.

MiKaPo 3.0 — 60 FPS real-time MMD motion capture

A ground-up rewrite of the solver and capture pipeline, authored by Claude (Fable 5). Headline: 60 FPS rendering with real-time capture — the render loop no longer shares a thread with detection.

Solver

• Data-driven core — one bone-definition table + generic direction/basis/twist solvers replace ~40 hand-written per-bone functions (~1,500 → ~500 lines). Verified bit-equivalent to the 2.0 solver on synthetic landmark suites, 1.7× faster, zero allocations per frame.
• reze-engine math — solver runs on Reze Engine's Vec3/Quat; matrix inversion replaced by quaternion conjugation; parent chains computed once per frame. Babylon.js remains only in the lazily-loaded debug skeleton preview.
• Roll witnesses — forearm/shin direction pins upper-arm/thigh roll, so elbow creases and knee planes orient correctly instead of being left to shortest-arc chance.
• Anatomical finger clamps — swing-twist decomposition per finger with human flexion/spread ranges; noisy landmark frames can no longer bend fingers backward.
• Visibility gating + hold-last-pose — occluded or off-frame limbs hold their pose instead of snapping to identity or chasing garbage.

Capture pipeline

• Web Worker detection — MediaPipe holistic runs off the main thread (frames shipped as transferred ImageBitmap), so WebGPU rendering holds 60 FPS during capture.
• Adaptive motion interpolation — pose tweens sized to the measured detection interval upsample ~30 Hz capture into smooth 60 FPS motion.
• Media-time smoothing — One-Euro filters run on video time; pause/seek resets cleanly instead of warping.
• Solve path runs outside React; per-model calibration happens once at load.

Face

• One-Euro smoothing per channel — fast blinks land at full depth.
• Morph alias resolution against each loaded model's morph list (e.g. ワ → にっこり → にやり), so more custom models work unmodified.

Live demo: https://mikapo.vercel.app

🤖 Generated with Claude Code

v2.6.0

• Update UI
• Add One-Euro filter
• Auto calibration to get rest reference pose

v2.5.0

• Use reze-engine v0.12.1 with anime aesthetic rendering
• Improve mediapipe and mmd loading indicators
• Reset morphs and bones upon new input image/video

v2.4.0

• Load user's local model

v2.3.0

• Add vmd recording and dump back
• Fix mediapipe version

v2.2.0

Add face morph back

v2.1.0 Migrate to Reze Engine

• Migrate from babylon-mmd to reze-engine

v2.0.1

• Use predetermined ratio for finger joints solver