Skip to content

11 ‐ Motion, Audio and Posturography Toolkit

Jump to bottom
Alexander Refsum Jensenius edited this page Jul 17, 2026 · 1 revision

Plain numpy/scipy signal-level toolkits for sound–motion–body analysis — no MgVideo/MgAudio object is required, just numpy arrays (position/pose/accelerometer trajectories, waveforms, sway paths, physiological waveforms). This page covers three complementary toolkits used across the ro, still standing, Westney-comparisons and cymbal-comparison studies (Jensenius): quantity of motion (band-limited QoM cores for position, pose and accelerometer data), audio features (scipy-only RMS/flux onsets, T60 decay, attack spectral centroid), and posturography & physiology (sway metrics, complexity/dynamics measures, respiration rate).

Quantity of motion (_qom)

Functions computing quantity-of-motion (QoM) signal cores from position, pose-landmark and accelerometer data — the unified signal cores behind band-limited QoM used across the stillstanding and Westney-comparisons studies. Call them directly on marker/landmark trajectories or accelerometer arrays.

Available functions

  • band_limited_qom(pos, fs, lo=0.3, hi=15.0, order=4, auto_decimate=True) — band-limited quantity of motion from a position trajectory: band-pass filter then per-frame speed. Automatically switches to a decimate+SOS regime for very low bands (e.g. 0.1–0.5 Hz postural sway) where a direct high-order band-pass would be fragile. Returns (speed, fs_out).
  • accel_to_speed(acc, fs, highpass=0.3, order=2, normalize_gravity=False) — integrated speed from a 3-axis accelerometer (the stillstanding study's "corpus method"): high-pass, integrate, high-pass again to kill drift.
  • group_qom(points, fs, lo=0.3, hi=15.0, **kwargs) — mean band-limited QoM (and mean speed envelope) over a group of markers/landmarks, e.g. head, shoulders, arms. Returns (qom, speed, fs_out).
  • pose_qom(landmarks, fs, lo=0.3, hi=5.0, **kwargs) — group_qom() tuned for 2-D MediaPipe pose landmarks (px/s), with a band that avoids landmark jitter. (For extracting the landmarks themselves, see Pose and Motion Capture.)
  • body_scale(landmarks, upper=(11, 12), lower=(23, 24)) — median torso length (shoulder-to-hip midpoint distance) as a body-size scale, for framing-invariant comparisons.
  • normalized_qom(landmarks, fs, scale=None, lo=0.3, hi=5.0, ...) — pose QoM divided by body_scale() (body-lengths per second) — invariant to camera framing/zoom.
  • grid_qom(frames, grid=(6, 4), region=(0.0, 1.0, 0.0, 1.0), threshold=8.0) — spatial-grid QoM from a stack of grayscale frames: thresholded inter-frame difference averaged per grid cell; returns per-cell time series plus a mean-motion heatmap (audience-region motion mapping).
  • envelope(x, fs, smooth=1.0, normalize=True) — Savitzky-Golay-smoothed, optionally z-scored envelope, for comparing motion/audio envelopes on a common scale.
  • bin_series(x, fs, bin_s=1.0) — mean of consecutive, non-overlapping bins (e.g. a per-second QoM envelope from a per-frame speed series).

Quick example

import musicalgestures as mg

# Body-scale-normalised pose QoM from MediaPipe landmarks (N, 33, 2), fps=30
qom, speed, fs_out = mg.normalized_qom(landmarks, fs=30.0)

# Band-limited QoM from a single marker's mocap trajectory (N, 3), 100 Hz
sway_speed, fs_out = mg.band_limited_qom(marker_xyz, fs=100.0, lo=0.1, hi=0.5)

# Per-second envelope for cross-modality comparison
env = mg.bin_series(sway_speed, fs_out, bin_s=1.0)

Audio feature extraction (_audiofeatures)

Scipy-only audio feature extraction for sound–motion analysis. These are plain functions on a mono waveform (y, sr) — they complement the librosa-based, figure-producing MgAudio methods documented on the Audio Analysis page with lightweight numeric outputs, and have no dependency on librosa. All onset detectors use the canonical peak-picker (pick_peaks(), see Pulse, Cycles and Alignment).

Available functions

  • rms_envelope(y, sr, window=0.02) — RMS energy envelope over consecutive, non-overlapping windows. Returns (env, rate).
  • spectral_flux(y, sr, nperseg=2048, noverlap=1536) — spectral-flux onset-detection function: positive first difference of STFT magnitude, summed over frequency. Returns (flux, times).
  • spectral_flux_onsets(y, sr, ..., threshold=None, min_interval=0.05) — onset times from spectral_flux(), peak-picked; threshold defaults to mean + 1 SD of the flux.
  • energy_onsets(y, sr, window=0.02, rel_threshold=0.15, min_interval=0.06) — onset times from the RMS envelope's half-wave-rectified first difference. Reliable for discrete strokes; over-fragments sustained rolls/tremolo.
  • t60_backward_decay(y, sr, window=0.02, spans=((-5, -35), (-5, -25)), rerise_db=6.0) — T60-style reverberation time by an ISO-3382-inspired level-span regression on the RMS envelope (no Schroeder backward integration). Returns (t60, span).
  • attack_spectral_centroid(y, sr, attack=0.12, nperseg=2048, hop=512, window=0.02) — energy-weighted mean spectral centroid over the first attack seconds after the RMS-envelope peak (e.g. discriminates strike placement/damping).

Quick example

import musicalgestures as mg
import soundfile as sf

y, sr = sf.read('/path/to/strike.wav')

onsets = mg.energy_onsets(y, sr, rel_threshold=0.15, min_interval=0.06)
t60, span = mg.t60_backward_decay(y, sr)
centroid = mg.attack_spectral_centroid(y, sr)

Posturography and physiology (_posture / _physio)

Pure numpy/scipy posturography (sway) and physiology signal features ported from the still standing study. These operate on plain arrays (a centre-of-pressure/marker path, a respiration waveform, ...) — no study-specific loaders or axis conventions.

Posturography (_posture)

Three families of measures on a 2-D sway path (or 1-D projection of one):

  • Sway amount / geometry — cop_sway_metrics(xy, t=None, fs=None, *, freq_band=(0.1, 5.0), resample_fs=50.0) computes path length/rate, 95% confidence-ellipse area, ML/AP ranges and SDs, and mean sway frequency; confidence_ellipse_area(xy, conf=0.95) and convex_hull_area(xy) give standalone area measures.
  • Control dynamics / complexity — stabilogram_diffusion(xy, fs, *, short_max_s=0.6, long_min_s=1.5, n_lags=40) (Collins-De Luca SDA: short/long-term Hurst exponents + crossover time); dfa(x, *, n_scales=18, min_scale=10) (detrended fluctuation analysis exponent); sample_entropy(x, m=2, r=0.2) (regularity/predictability); spectral_edges(x, fs, *, edges=(0.5, 0.95)) (median + 95% spectral-edge frequency); sway_texture(speed, fs, *, frozen_threshold=2.0) (frozen fraction + burst rate); principal_axis_projection(xy) (1-D PCA reduction feeding the above).
  • Direction / extent — sway_orientation(xy) (principal sway-axis angle + anisotropy); axial_rayleigh(angles_deg) (Rayleigh test for a shared preferred sway axis across sessions); spatial_extent(pos, fs, *, ellipse_conf=0.95, window_s=20.0, vertical_axis=None) (dispersion radius, confidence-ellipsoid volume, drift ratio).

Physiology (_physio)

  • respiration_rate(waveform, fs, *, band=(0.1, 0.6), window_s=30, step_s=30) — windowed breathing rate (breaths/min) via band-pass + per-window Welch spectral peak. Returns {"rate_bpm", "times_s", "median_bpm"}.
  • spectral_band_fractions(signal, fs, bands, *, total_band=(0.1, 8.0), nperseg_s=20) — fraction of a signal's Welch power in each caller-supplied named band (e.g. {"cardiac": (0.9, 1.3), "resp": (0.12, 0.5)}) relative to total_band — the generic "cardiorespiratory QoM artifact" diagnostic, with no built-in dependence on a particular sensor.

Quick example

import musicalgestures as mg

# Centre-of-pressure sway metrics, xy shape (T, 2) = [ML, AP], 50 Hz
metrics = mg.cop_sway_metrics(cop_xy, fs=50.0)
sda = mg.stabilogram_diffusion(cop_xy, fs=50.0)
alpha = mg.dfa(mg.principal_axis_projection(cop_xy))

# Breathing rate from a chest-belt waveform, 25 Hz
resp = mg.respiration_rate(breathing_waveform, fs=25.0)

# How much of a chest-accelerometer QoM signal is cardiac vs. respiration artifact
fractions = mg.spectral_band_fractions(
    chest_qom, fs=25.0, bands={"cardiac": (0.9, 1.3), "resp": (0.12, 0.5)})

Provenance

Sources: stillstanding study and Westney-comparisons study (Jensenius) for _qom — band_limited_qom() unifies the band-limited QoM cores used on mocap markers (mm), MediaPipe landmarks (px) and slow postural sway across both studies; accel_to_speed() is the stillstanding study's chest-accelerometer "corpus method"; body_scale()/normalized_qom() come from the Westney-comparisons study's framing-invariant with/without-audience comparison of a pianist's motion; grid_qom() is the Westney-comparisons study's audience-region motion mapping in a concert hall.

Sources: cymbal-comparison study and Westney-comparisons study (Jensenius) for _audiofeatures — rms_envelope(), energy_onsets(), t60_backward_decay() and attack_spectral_centroid() reimplement the cymbal-comparison paper's method descriptions (their numeric defaults are PROVISIONAL — tune per dataset); spectral_flux()/spectral_flux_onsets() come from the Westney-comparisons study.

Source: still standing study (Jensenius) for _posture/_physio — posturography and micromotion analyses of the international "standstill" championships and related datasets (balance, sway-dynamics, sway-texture, sway-direction and spatial-range reports; Deichman/Equivital respiration and cardiorespiratory-artifact analyses). The from-scratch SDA/DFA/sample-entropy implementations are validated against known-answer synthetic signals in the test suite.

API docs

A project from the fourMs Lab, RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion, Department of Musicology, University of Oslo.

Clone this wiki locally