12 ‐ Pose and Motion Capture

Three complementary sources of pose and motion-capture data for downstream signal analysis (quantity of motion, alignment, event detection — see the other Sound–Motion Analysis Toolkit pages): rendered skeleton video/plots via MgVideo.pose() and friends, markerless landmark trajectories via _posetools, and marker-based motion capture I/O via _mocap. The rendering methods return MgVideo/MgFigure/MgImage objects for visualization; _posetools and _mocap hand back plain numpy trajectory arrays for further analysis rather than a rendered output.

---

Pose rendering (MgVideo.pose())

All pose-rendering methods are called on an MgVideo object and return a result you can display or chain into further methods (for the complete MgVideo method index see Video Analysis).

• pose() — skeleton estimation; MediaPipe (default; fast on plain CPU, 33 landmarks) or OpenPose (multi-person); optional per-marker motion trails (marker_history) (returns MgVideo)
• pose_waterfall() — 3D spatio-temporal waterfall of pose markers in (x, time, y) space; style='trajectories' (default, continuous per-marker paths), 'markers', 'skeleton', or 'both' (markers/skeleton at sampled time slices); axes=False for a clean render, crop=True to tighten to the data and trim whitespace; a pose-based counterpart to silhouette_waterfall() (returns MgFigure)
• pose_segments() — circular (polar rose) motion plots + circular statistics (mean angle, resultant length R, circular std, range of motion, mean angular speed) for each body segment (the bone between two joints); saves a stats CSV. Reuses cached pose keypoints from a prior pose() call, otherwise runs pose() first (returns MgFigure)
• pose_center() — centre the pose data on its global centroid (a 2D port of the MoCap Toolbox mccenter) (returns MgFigure)
• pose_distance() — per-marker cumulative distance travelled plus the average across markers (a 2D port of the MoCap Toolbox mccumdist) (returns MgFigure)

pose_waterfall() renders a 3D spatio-temporal waterfall of the pose markers flowing through (x, time, y) space, and pose_segments() draws a grid of polar rose plots — one per body segment — of its orientation-angle distribution.

Quick example

import musicalgestures as mg

mv = mg.MgVideo('/path/to/video.avi')

mv.pose().show()                                       # skeleton video (MediaPipe by default)
mv.pose_waterfall(style='trajectories').show()         # 3D spatio-temporal waterfall
mv.pose_waterfall(style='skeleton', crop=True).show()  # sampled skeleton slices, cropped
mv.pose_segments(n_bins=24, cmap='magma', ncols=4).show()  # per-segment circular statistics
mv.pose_center().show()                                # centre pose data on global centroid
mv.pose_distance().show()                               # per-marker cumulative distance

---

Landmark-trajectory extraction (_posetools)

The array-level counterpart to the rendering-oriented MgVideo.pose() pipeline above: use this module for plain numpy trajectories to feed into QoM/alignment/event-detection, rather than a rendered skeleton video.

• extract_pose_landmarks(filename, fps=None, width=None, model_complexity=1, world_landmarks=False, min_detection_confidence=0.5, min_tracking_confidence=0.5, max_frames=None, target_name=None, quiet=True, verbose=True) — runs MediaPipe Pose over a whole video via an FFmpeg decode pipe and returns tidy per-landmark trajectories (33 MediaPipe landmarks). Needs the optional mediapipe dependency (pip install musicalgestures[pose]), imported lazily. Returns a dict with time, landmarks (F, 33, 3) = (x_px, y_px, visibility), optional world, detected, detection_rate, fps, width, height, names; pass target_name to also write a tidy CSV.
• midpoint(a, b) — element-wise midpoint of two landmark trajectories, e.g. the shoulder midpoint as an upper-torso proxy point.

For pose-derived quantity-of-motion metrics (pose_qom(), body_scale(), normalized_qom()), see the Quantity of motion section of the Motion, Audio and Posturography Toolkit page.

Pose-derived signals (limb speed, impacts)

• limb_speed_from_landmarks(xy, confidence, fps, conf_gate=0.5, merge='max_lr', smooth_taps=3) — confidence-gated image-plane speed of one or more candidate limbs (e.g. left/right wrist), merged by element-wise maximum so motion of either limb registers. Numpy-only — works with landmarks from MediaPipe, OpenPose, YOLO-pose or mocap.
• impact_events(pos_by_point, fps, rel_thresh=0.12, min_interval_s=0.10) — candidate impact events from point trajectories via double-differentiated (acceleration) peaks, bilateral max across candidate points. Returns {"index", "time", "magnitude", "accel"}.

Quick example

# Markerless video pose: extract landmarks, then derive limb speed and impacts
result = mg.extract_pose_landmarks('/path/to/video.mp4', fps=25, width=480)
wrists = result['landmarks'][:, [15, 16], :2]           # left/right wrist xy
conf = result['landmarks'][:, [15, 16], 2]               # visibility
speed = mg.limb_speed_from_landmarks(wrists, conf, fps=result['fps'])
impacts = mg.impact_events(wrists, fps=result['fps'])

---

Mocap I/O (_mocap)

• read_qtm_tsv(path) — robust reader for Qualisys Track Manager (QTM) TSV exports: locates the MARKER_NAMES header, autodetects the numeric data block, converts exact-zero XYZ gap-fills to NaN, falls back from UTF-8 to latin-1. Returns (marker_names, data, fs) with data of shape (T, M, 3).
• compare_modality_envelopes(env_a, env_b, fs_a, fs_b) — resample two precomputed motion envelopes onto a common one-second grid and correlate them, e.g. to validate a video-pose QoM envelope against a mocap QoM envelope. Returns {"r", "n"}.
• dominant_frequency(x, fs, band=(0.3, 4.0)) — dominant Welch spectral peak of a signal within a band (e.g. a body-part speed or vertical-position signal). Not re-exported at the top level (it would shadow musicalgestures.dominant_frequency from _analysis) — call it as musicalgestures._mocap.dominant_frequency(...).

Quick example

names, data, fs = mg.read_qtm_tsv('/path/to/session.tsv')   # data: (T, M, 3)
agreement = mg.compare_modality_envelopes(mocap_env, video_env, fs_a=fs, fs_b=30.0)

Provenance

The pose-rendering MgVideo methods (pose(), pose_waterfall(), pose_segments()) are core toolbox features, not tied to a specific study; pose_center() and pose_distance() are 2D ports of the MoCap Toolbox's mccenter/mccumdist.

_mocap and _posetools are sourced from the still standing and Westney-comparisons studies (Jensenius): read_qtm_tsv() consolidates several near-duplicate QTM loaders copy-pasted across the study scripts; compare_modality_envelopes() is the MediaPipe-vs-mocap validation from the still standing study. _posetools is consolidated from the author's study extractors (stillstanding's mp_extract_westney.py/pose_motion.py and the Westney-comparisons study's concert_mediapipe.py/reh_pose.py/a1_labstage.py) plus the cymbal-comparison study's markerless striking-wrist speed and kinematic impact detection (reimplemented from the paper's method description; the numeric defaults are that study's PROVISIONAL values for 120 Hz mocap hand data — tune per dataset).

API docs

• _mocap API reference · source
• _posetools API reference · source
• Video Analysis on the docs site — full MgVideo reference, including pose() and related rendering methods