Skip to content

13 ‐ Working with ambiscape

Jump to bottom
Alexander Refsum Jensenius edited this page Jul 27, 2026 · 3 revisions

MGT-python and ambiscape are sister toolboxes from the fourMs lab: MGT owns the pixels, ambiscape owns the samples. MGT does motion analysis, pose, visualization, and 360-degree video handling; ambiscape does soundscape levels, spatial audio, sound-event taxonomies, and long-duration session analysis. This page documents how to use them together on one recording session — the shared time base, the audio bridge, device synchronization, and the per-camera 360 workflow.

The ambiscape-side version of this page is Working with MGT-python in the ambiscape wiki.

The division of labour

Question Toolbox
How did people/objects move? (QoM, motiongrams, pose, optical flow) MGT
How does the place sound? (Leq, events, diffuseness, Schafer/Schaeffer) ambiscape
Stitch/flatten 360 video (Insta360, GoPro MAX, Theta S, Garmin RAW) MGT
Ingest camera audio, pick the ambisonic track, spatial audio analysis ambiscape
Align recordings from different devices MGT (align_recordings) measures; ambiscape (calibration.json) applies
One combined summary of an audio-visual session both, via namespaced keys in one summary.json

MGT never becomes a dependency of ambiscape (its core stays numpy/scipy/soundfile-light). The one crossing point lives in MGT:

pip install "musicalgestures[soundscape]"   # adds ambiscape

The shared time base

Everything joins on the wall clock. Both toolboxes parse recording start times from filename stamps with byte-identical rules: a YYYYMMDD_HHMMSS or YYMMDD_HHMMSS stamp anywhere in the filename (phones, recorders, and 360 cameras name files this way out of the box). ambiscape additionally reads BWF bext timestamps in WAVs; MGT falls back to file mtime.

from musicalgestures._timecode import filename_datetime, media_start_datetime

filename_datetime("20260724_142731_00_037.insv")   # datetime(2026, 7, 24, 14, 27, 31)

Any MgFeatures container whose metadata carries a start_datetime (a datetime or ISO string) can place every sample on the wall clock:

mgf.metadata["start_datetime"] = media_start_datetime(video_path)
t_epoch = mgf.absolute_times()          # epoch seconds per sample

Times are naive local time on both sides — consistent as long as all devices lived in one timezone.

Device synchronization

Device clocks are always slightly wrong (a real five-device session showed offsets of −5 s, +2 s, and 0 s). Measure the offset between two recordings of the same scene by envelope cross-correlation:

from musicalgestures._sync import align_recordings

res = align_recordings("insta360_00.insv", "voicerecorder.m4a")
# {'lag_s': 17.125, 'peak': 0.68}   # positive = second file starts later

peak below ~0.3 means the alignment is unreliable (too little shared audio). Feed the measured offsets to ambiscape's per-take calibration — calibration.json in the session folder:

{
  "clock_offsets_s": {"20260724_143753_insta039.wav": -5.0,
                      "20260724_143810_samsungaud.wav": 2.0}
}

ambiscape warns about keys that match no file in the session (a typo would otherwise silently misalign the analysis).

The audio bridge: ambiscape features as MgFeatures

soundscape_features() opens an ambiscape session (a folder of WAVs on one clock), runs or reuses its cached feature extraction, and returns the 1 Hz series as an MgFeatures on the wall clock:

from musicalgestures._soundscape import soundscape_features

aud = soundscape_features("2026-07-24-belfort-session")
aud.feature_names            # ['aud_level_db']
aud.absolute_times()         # epoch seconds, day0-midnight anchored

Because both sides expose absolute_times(), joining motion and audio is one merge:

import pandas as pd
motion_df = pd.DataFrame({"t": mot.absolute_times(), "qom": mot["qom"]})
audio_df = pd.DataFrame({"t": aud.absolute_times(),
                         "level": aud["aud_level_db"]})
joint = pd.merge_asof(motion_df.sort_values("t"), audio_df.sort_values("t"),
                      on="t", tolerance=0.5)

And the combined-summary convention: fold motion medians/IQRs into the same summary.json that ambiscape writes, using the mot_ prefix (mirroring ambiscape's vis_ prefix for visual features):

from musicalgestures._soundscape import merge_into_summary

merge_into_summary(mot, "session/analysis/summary.json")   # mot_qom_median, ...

Namespaces: mot_ = MGT motion, vis_ = ambiscape vision, aud_ = audio series exposed inside MGT.

360 cameras: who does what

Camera Video (MGT) Audio (ambiscape ≥0.19)
Insta360 X3/X4 (two .insv lens files) stitch_dual_fisheye(front, back) or Mg360Video.from_dual_fisheye(); calibrate_dual_fisheye_fov() finds the true lens FOV from a probe frame drop the .insv in a session folder — ingests directly. Caveat: Insta360 4-channel audio is not B-format ambisonics; treat spatial metrics from it as unreliable
GoPro MAX / MAX2 (.360, custom EAC) flatten_gopro360(path) — probe-driven (also accepts chunk-merged .mkv); MAX2 resolutions are scaled and experimental ingests the 4-channel AmbiX PCM track automatically (best-stream selection beats the stereo AAC), cached at 24-bit
Ricoh Theta S (legacy dual-fisheye .MP4) flatten_theta360(path) — explicit invocation (the rotated lenses defeat plain v360); fov_deg/roll_deg tunable mono audio; ingests as-is
Garmin VIRB 360 (.MP4/.GLV) already equirectangular — nothing to flatten; RAW-mode 200° hemisphere pairs go through stitch_dual_fisheye 4-channel AmbiX with an empty Z: azimuth valid, elevation meaningless; prefer in-camera originals over VIRB Edit exports

Notes that save an afternoon: Insta360 .lrv proxies carry the same-bitrate audio as the .insv (fine for audio-only work); GoPro .lrv is plain dual-fisheye; iPhone spatial captures hold a 4-channel APAC track no non-Apple tool decodes — transfer originals and decode via afconvert on macOS 15+ before ambiscape can use it. GoPro and Theta S flattening are validated against real camera footage (the AMBIENT lab's 2023 four-camera comparison recordings): GoPro output is verified upright with continuous wrap, and Theta output matches the RICOH THETA app's own equirect export up to a yaw offset. MAX2 remains probe-scaled and experimental until a real MAX2 file is tested.

A worked example

The Belfort (Bruges) session, 2026-07-24 — five devices, 36.5 minutes: iPhone video, Samsung 4K video + voice recorder, Insta360 in two rooms.

  1. Align: align_recordings on the overlapping pairs anchored everything to the network-synced phone clock (Insta360 −5 s).
  2. Video: stitch_dual_fisheye flattened both Insta360 recordings (calibrated FOV: 195°, not the nominal 190°).
  3. Audio: one ambiscape session over all five extracted tracks; escapement measured the tower clock's 1.649 s beat (±11 ms wander), carillon --events transcribed 29 distinct bell strike notes, taxonomy rendered the Schafer timeline and Schaeffer map.
  4. Joint: audio events time-stamped on the same clock as the flattened video — finding the drum-chime moment in the 360 footage is a timestamp lookup, not a search.

See also

A project from the fourMs Lab, RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion, Department of Musicology, University of Oslo.

Clone this wiki locally