Skip to content

Working with MGT python

Jump to bottom
Alexander Refsum Jensenius edited this page Jul 26, 2026 · 3 revisions

ambiscape and the Musical Gestures Toolbox for Python (MGT) are sister toolboxes: ambiscape owns the samples, MGT owns the pixels. ambiscape analyses the sound of a place; MGT analyses motion in video, handles pose, and — since 2026 — flattens and stitches every 360-camera format the lab uses. Together they cover an audio-visual field session end to end. This page documents the seams: the shared clock, device synchronization, the one-directional bridge, and the per-camera workflow.

The MGT-side version of this page is Working with ambiscape in the MGT wiki.

Design rules (why the integration looks like this)

  1. ambiscape never imports MGT. The core stays numpy/scipy/soundfile-light so it runs in course CI and on thin machines. The single crossing point is on MGT's side, behind an optional extra: pip install "musicalgestures[soundscape]".
  2. One time contract. A recording's start time comes from a filename stamp (YYYYMMDD_HHMMSS / YYMMDD_HHMMSS, identical regexes in both toolboxes), a BWF bext chunk (ambiscape), or file mtime (last resort). Feature series carry their absolute start, so audio and motion join on the wall clock with a plain merge_asof.
  3. Namespaced summaries. One analysis/summary.json per session: ambiscape's descriptors, vis_* keys from ambiscape vision --merge, and mot_* keys folded in by MGT's merge_into_summary(). No second dialect.

Multi-device sessions: measure with MGT, apply in ambiscape

Real sessions have per-device clock errors (a five-device belfry session: −5 s, +2 s, 0 s). MGT measures pairwise offsets by envelope cross-correlation:

from musicalgestures._sync import align_recordings
align_recordings("insta360_00.insv", "recorder.m4a")
# {'lag_s': 17.125, 'peak': 0.68}  # positive = second starts later; peak <0.3 = distrust

ambiscape applies them per take via calibration.json in the session folder:

{
  "clock_offset_s": 0.0,
  "clock_offsets_s": {"20260724_143753_insta039.wav": -5.0,
                      "20260724_143810_samsungaud.wav": 2.0}
}

Keys that match no session file trigger a warning — a typo would otherwise silently misalign exactly the multi-device analysis this exists for. Field-side prevention beats both: clap once while all devices roll (see Field Recording Protocol).

What ambiscape ingests directly (v0.19+)

Session folders accept camera containers, not just WAVs: .mp4 .mov .m4a … .insv .lrv .glv .360 decode on ingest via ffmpeg. Two behaviours matter for 360 cameras:

  • Best-stream selection: containers with several audio streams decode the best decodable one — most channels wins. A GoPro MAX .360 therefore yields its 4-channel first-order AmbiX PCM track (verified B-format in the lab's spatial-audio study), not the stereo AAC sitting in front of it. Deep PCM sources cache at 24-bit.
  • Camera caveats (from Riaz, Guo & Jensenius): Insta360 4-channel audio is not B-format — ambiscape will still label 4-channel input ambix by count, so treat Insta360 spatial metrics as unreliable until that layout is established. Garmin VIRB 360 is AmbiX with an empty Z channel ("planar"): azimuth fine, elevation meaningless — and prefer in-camera originals over VIRB Edit exports (bugged FOA track). iPhone spatial captures carry an APAC track ffmpeg cannot decode — ambiscape uses the stereo AAC unless you decode the APAC to 4-channel PCM on Apple hardware first (afconvert, macOS 15+).

Video-side prerequisites (MGT)

Flattening/stitching lives in MGT, one call per camera family:

Camera Call
Insta360 two-lens .insv pairs stitch_dual_fisheye(front, back) (auto FOV calibration; a lab X4 measured 195°, not the nominal 190°)
GoPro MAX/MAX2 .360 (custom EAC) flatten_gopro360(path) — probe-driven, accepts chunk-merged .mkv; MAX2 experimental
Ricoh Theta S legacy dual-fisheye flatten_theta360(path) — explicit call; rotated lenses defeat plain v360
Garmin VIRB 360 nothing (already equirectangular); RAW 200° pairs → stitch_dual_fisheye

GoPro and Theta S flattening are validated on real footage (the lab's 2023 four-camera comparison data); Theta output matches the THETA app's export up to a yaw offset.

The bridge back: ambiscape features inside MGT

For joint analysis, MGT pulls an ambiscape session's 1 Hz features into its own container type, wall-clocked and ready to merge with motion series:

from musicalgestures._soundscape import soundscape_features, merge_into_summary

aud = soundscape_features("2026-07-24-belfort-session")   # MgFeatures, aud_level_db
merge_into_summary(motion_features, "…/analysis/summary.json")  # adds mot_* keys

This runs (or reuses) ambiscape.features.extract_session under the hood — the same cache ambiscape analyze builds, so nothing is computed twice.

A worked example

The Belfort (Bruges) session, 2026-07-24: five devices, 36.5 min, two rooms.

  1. MGT's align_recordings fixed the device clocks; offsets went into calibration.json.
  2. One ambiscape session over all five tracks: analyze → descriptors; escapement → the tower clock's 1.649 s beat, ±11 ms wander; carillon --events → 29 distinct strike notes across the quarter chimes; taxonomy → Schafer timeline + Schaeffer map.
  3. MGT stitched the Insta360 pairs to equirectangular panoramas on the same clock — every transcribed bell strike indexes straight into the 360 footage.

See also

Clone this wiki locally