12 ‐ Segmentation and Annotation

For complete documentation see the API reference for _tracks, _voice, _laughter, _actions, _hierarchy, _annotate, _views and _zoomview.

---

Annotating a recording means writing down what happens and when: this phrase starts here, this laugh belongs to that exchange. The tools on this page prepare that work —cutting hours into candidate pieces, proposing where speech and laughter are—and then read the finished annotations back in, so a person's judgements and the measured signals can be analysed together.

Most of this toolbox turns a video into a picture. This page is about the other thing: turning a long recording into material a person can annotate, and then looking at what they annotated.

It exists because hours of video defeat the usual approach. You cannot scrub a two-and-a-half-hour session looking for the interesting parts, and a motiongram of one is a thin smear.

1. Extract once, read many times

extract_tracks walks the file once and writes quantity of motion and both motiongram axes (row and column means of the motion frame) into preallocated memory-mapped files, then one more ffmpeg pass writes the true videograms (means of the picture) beside them; read_columns builds the pyramid levels it needs on first use, so any time range can then be read at any width without touching the video again. The returned dict (and tracks.json) carries analysis_dir, the folder everything landed in.

from musicalgestures._tracks import extract_tracks, build_pyramid, check_tracks

extract_tracks('session.mp4')                 # writes into session_analysis/
build_pyramid('session_analysis')
check_tracks('session_analysis')
# {'preallocated': 475680, 'last_nonzero': 475679, 'marker_gaps': [], 'complete': True}

Always call check_tracks rather than looking at file sizes. The files are preallocated, so their length is what the run intended, not what it achieved. A run killed halfway leaves a full-length file of mostly zeros.

2. Find where things are

import soundfile as sf
from musicalgestures._voice import speech_segments
from musicalgestures._laughter import laughter_segments

y, sr = sf.read('session_audio16k.wav')
speech = speech_segments(y, sr=sr)            # where somebody is speaking
laughs = laughter_segments(y, sr=32000)       # where laughter probably is

Two cautions, both learned the hard way.

Check the channels before making a file mono. ffmpeg -ac 1 averages them, which is right for a stereo pair and wrong for two unrelated inputs. Averaging a room microphone with a dead channel halves the level; on one recording that cost twenty-eight times the detected speech.

laughter_segments returns proposals, not findings, and its scores sit low—a threshold of 0.6 can return nothing at all from a recording full of laughing. Take the threshold from the distribution and keep the score track.

3. Cut motion into spans, and build the levels

from musicalgestures._actions import segment_actions
from musicalgestures._hierarchy import Hierarchy

phrases = segment_actions(qom, fps, threshold=0.15, min_duration=3.0,
                          range_mode='robust', range_percentiles=(50., 99.))
h = Hierarchy(levels={'motion': phrases, 'speech': speech, 'laughter': laughs})

range_percentiles=(50., 99.) measures the range from the median up rather than from the 1st percentile. The threshold is a fraction of motion above rest, so the bottom of the range has to be rest—and the quietest one per cent usually is not, because sensor noise and anything moving in shot put a floor under everything.

4. Export for a person

from musicalgestures._annotate import to_elan

to_elan(h, 'session.mp4', 'session.eaf', nest=False,
        vocabularies={'quality': ['light', 'heavy', 'sudden', 'sustained']})

nest=False gives independent tiers. Nesting is right for a hierarchy—actions inside phrases inside parts—and wrong for layers that merely coincide: speech is not inside motion. vocabularies gives the annotator a drop-down, which is what stops ENJOYMENT, Enjoyment and enjoyment becoming three categories.

5. Look at what is there

from musicalgestures._views import filmstrip, concordance, tier_map
from musicalgestures._zoomview import zoomable_page

tier_map(h, duration_s, 'tiers.png')                    # where is anything to look at
filmstrip('session.mp4', 7500, 8000, n=12, hierarchy=h, out='strip.png')
concordance('session.mp4', laughs, 'laughs.png', n_cols=8)
zoomable_page('session_analysis', duration_s, 'zoom.html', hierarchy=h,
              audio='session.wav',                       # waveform + spectrogram band
              video={'videogram': vg, 'motiongram': mg})  # switchable strips

concordance is the one with no equivalent elsewhere in the toolbox: it puts every instance of a category side by side. Coding 183 of something one at a time, hours apart, is how a category drifts.

zoomable_page writes a single offline HTML file. Scroll to zoom, drag to pan, 0 to reset. It says on the picture when you have zoomed past the data it contains. audio= adds a band that switches between waveform and spectrogram on the same clock; video= takes named (rows, time) arrays—a videogram and a motiongram, say—with a switch when there is more than one; and player= puts the recording itself above the strips, referenced by relative name so the page still needs no server—click the timeline to seek, and a playhead crosses every band. For any single video, MgVideo.zoompage() builds the whole page in one call, extraction and all.

6. What in the recording is not the person you are studying

from musicalgestures._plate import room_plate, occupancy_track, restless_regions

plate, _ = room_plate('session.mp4')              # the empty room
idx, occ = occupancy_track('session.mp4', plate)  # how much of the frame is a person
mask = restless_regions(stack)                    # pixels that change regardless

room_plate takes the per-pixel median over sampled frames, not the mean: a mean keeps a faint ghost of everyone who crossed, and subtracting a ghost leaves holes shaped like people. Occupancy then answers what quantity of motion cannot—somebody standing still has no motion and plenty of occupancy.

restless_regions marks what changes whether or not anyone is there: a screen showing a video call, a window, somebody sitting at a table. On one corpus that was 2.8 to 7.1 per cent of all measured motion. Median absolute deviation rather than range, because a screen changes in nearly every frame while a dancer occupies a pixel occasionally, and a range marks both alike—which would mask the dancer along with the distraction.

7. Bring somebody else's annotations in

from musicalgestures._elan import read_elan_csv
from musicalgestures._alignment import align_by_audio, envelope_from_audio

theirs = read_elan_csv('their_export.csv')       # keeps which media it describes
offset, r, agree, tried = align_by_audio(their_env, our_env, fs=20.0)

read_elan_csv keeps the provenance line naming the file the times belong to, because taking the numbers and discarding that is how annotations land silently on the wrong timeline. align_by_audio finds where a short recording sits inside a long one, which is what puts them right.

8. Describe the room itself

from musicalgestures._soundscape import soundscape_features
F = soundscape_features('session.mp4')     # a video is a valid input

Level, spectral centroid, flatness, and ten octave bands at 1 Hz, so they join a motion series. Worth doing before comparing two recordings: on one corpus, two rooms recording the same event differed by 17–22 dB in the octave where speech lives, which no amount of level normalisation recovers.

What to distrust

• File length, for anything preallocated or still being written.
• A threshold that looks reasonable. Take it from the distribution and say what you took.
• A self-similarity matrix you have not tested. structure_map always produces a plausible picture; check it against something you already know before believing it. Its docstring carries the numbers from when it did not work.
• An offset between two rooms that hear each other. That is a clock offset plus a network latency, and the two separate: the correlation has two peaks, whose midpoint is the offset and whose separation is the round trip.