Test fixtures to investigate technical challenges of IIIF-AV
- Gapless audio playback. One IIIF canvas with three annotations with audio content, that must be played in sequence with no gap.
- Gapless video playback. One IIIF canvas with three annotations with video content, that must be played in sequence with no gap.
- Synchronised video example. Videos playing at the same time. The source videos contain markers to demonstrate the synchronisation points. One IIIF canvas annotated with three videos.
- Synchronised audio and video example. Audio soundtrack aligned to playing video.
- Video synchronised with gapless audio, text that includes gapless join of two audio tracks. Timed text annotations appear throughout including at the same time as the gap transition
- Fire. Images, video and text
- Mahler 3 from Indiana showing range navigation. There are two canvases in this example, each representing a CD. The range metadata provides the navigation.
- Lunchroom manners from Indiana showing range navigation in a video.
- Bach: Brandenburg Concerto no. 1 from the British Library showing range navigation of audio.
Later - verify approach works with mixture of file formats, and MPEG-DASH/HLS
See https://github.com/IIIF/iiif-av/blob/master/source/api/av/index.md for model discussion.
Two additional useful fixtures:
- simplest time-based canvas - Shows an image for part of the duration of a canvas.
- parts of media annotated onto different canvas - Example - divide a performance up into scenes on different canvases.
based on Evaluation of Existing Approaches
UI methods (canvas navigation, range navigation, logging, helpers)
Parser and media initialisation methods
Canvas clock methods (player initialisation, timing, synchronisation)
Canvas Instances / Canvas Clock
The canvas clock controls synchronisation and playback of media items. It is an abstract timing mechanism, which serves as a point of reference to the synchronisation modules, rather than being based on the native timing mechanisms of a media element. The
canvasClockUpdater()method updates the abstract timing (based on
canvasClockTime). Every canvas instance has its own canvas clock, player element and updater methods (see Media Synchronisation). A canvas clock is always independent of any other canvas instances (playback state, current time, etc.) and contains its own set of media elements (see Manifest Parsing).
A single canvas instance (see global object
canvasInstances) contains the following properties and methods:
- data (object)
- canvasClockDuration (number)
- canvasClockStartDate (number)
- canvasClockTime (number)
- canvasWidth (number)
- canvasHeight (number)
- playerElement (DOM element)
- mediaElements (array) -> see Manifest Parsing
- pauseCanvas (function)
- playCanvas (function)
- setCurrentTime (function)
- updateMediaActiveStates (function)
- checkMediaSynchronization (function)
- synchronizeMedia (function)
- playbackStalled (function)
- canvasClockUpdater (function)
- highPriorityUpdater (function)
- lowPriorityUpdater (function)
- isPlaying (boolean)
- wasPlaying (boolean)
- isStalled (boolean)
- stallRequestedBy (array)
initContents()parses all media items from the manifest data, pushes them to the custom
mediaElementsarray of the respective canvasInstance (see
canvasInstances.mediaElements) and renders them in the DOM (both within the canvas and inside the timeline). The
mediaElements.elementcontains a binding to the rendered DOM element.
- A single media element object contains the following properties and methods:
- type (string)
- source (string)
- start (number)
- startOffset (number)
- end (number)
- endOffset (number)
- top (number)
- left (number)
- width (number)
- height (number)
- element (DOM element)
- timelineElement (DOM element)
- checkForStall (function)
- timeout (function)
- active (boolean)
- outOfSync (boolean)
- Based on the canvas clock timing, media items are synchronised using a set of updater methods. These methods are controlled by two intervals (using
window.setInterval) of different frequencies (
highPriorityFrequency). The intervals are set and cancelled by the canvas instance methods
pauseCanvas(). Media items are synchronised in two ways:
updateMediaActiveStates()controls the visibility of items (based on the
mediaElements.endproperties, as well as the canvas and media element timing offsets)
currentTimeof video and audio elements relative to the canvas clock time, as well as their playback state (
- Proper synchronisation is regularly checked and corrected in
checkMediaSynchronization(). Additionally, every media item has its own
mediaElements.checkForStall()method, which reacts to changes in the
readyStateof video and audio elements (buffering or loading issues). This method is coupled with the
checkMediaSynchronization()logic, as media buffering is the main cause for synchronisation lags.