Permalink
Branch: master
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
105 lines (81 sloc) 9.55 KB

Media Synchronisation and Playback -
Evaluation of Existing Approaches

Introduction

The goal of this document is to give an overview of technical challenges and state of the art implementations. The document covers web standards and javascript frameworks / libraries. The different approaches are summarized, compared and evaluated regarding their use in the proof of concept implementation of the IIIF A/V test fixtures.


Web Standards

  • Synchronized Multimedia Integration Language (SMIL 3.0)
    https://www.w3.org/TR/SMIL3/
    • Last updated: 01 December 2008
    • Allows describing the temporal behavior of a multimedia presentation, associate hyperlinks with media objects and describe the layout of the presentation on a screen in XML. No relevant recent client implementations, included for for the sake of completeness and reference to a timing model which has been crafted in detail over many years.
    • Relevant sections:
  • SMIL Timesheets 1.0
    https://www.w3.org/TR/timesheets/
    • Last updated: 28 March 2012
    • Makes SMIL 3.0 element and attribute timing control available to a wide range of other XML-based languages. "SMIL Timesheets can be seen as a temporal counterpart of CSS".
    • Relevant sections:
  • HTML5 Media Element (Video & Audio)
    https://www.w3.org/TR/html5/embedded-content-0.html#media-element
    • Last updated: 28 October 2014
    • Native integration of video and audio elements in HTML5 documents. Includes a JS API and Media Events (onPlay, onEnded, timeupdate, etc.). There is however no native event system which is able to capture / fire events at certain points of time (this still requires a loop which runs every time the timeupdate event is fired and checks a list of start and end times in order to synchronize contents).
    • Relevant sections:
    • See also:
  • Timing Object (Draft Community Group Report)
    http://webtiming.github.io/timingobject/
    • Last updated: 30 November 2015
    • [...] "local object that may be used by Web clients to ensure precisely timed operation as well as flexible timing control. If multiple timing-sensitive components take direction from the same timing object, their behaviour will be precisely aligned in time (synchronized)". Can be connected to a web service distributing timing information to one or more devices (allows multi device synchronisation). Very promising approach towards a media independent timing / clock standard. Currently in draft state, worked on by a small W3C community group (no signs at this point that W3C would officially support these efforts in their roadmap).
    • Relevant sections:
  • Media Source Extensions (MSE)
    https://www.w3.org/TR/media-source/
    • Last updated: 17 November 2016
    • Allows using Javascript to generate media streams (for playback as HTML5 media element). As these streams can be of various media types, MSEs could be seen as a time container which is used to generate a media stream and is optimized for seamless playback (the result is a conventional media element with the same API and events as a video element). There are however no relevant client implementations dealing with that use case. The main use case for MSEs is adaptive streaming.
    • Relevant sections:
    • See also:

Frameworks / Libraries

  • MediaGroup JS
    https://github.com/rwaldron/mediagroup.js

    • Last updated: 15 September 2011
    • JS implementation of the mediagroup attribute
      (see HTML5 Media Element > mediagroup attribute & MediaController)
  • Timesheets JS
    https://github.com/timesheets/timesheets.js

    • Last updated: 29 January 2013
    • JS implementation of SMIL Timesheets. Allows defining in / out times or durations directly in HTML elements (as attributes).
    • See also:
  • Popcorn JS
    https://github.com/mozilla/popcorn-js

    • Last updated: 6 June 2015 (now officially abandoned by Mozilla)
    • HTML5 Media Framework by Mozilla. Includes a sophisticated event system and plugin architecture, Managing track events after initialisation is very complicated and in some cases not possible. The HTMLNullVideoElement provides an abstraction of the native HTML5 media element API (similar to the IIIF A/V CanvasClock concept).
    • See also:
  • timingsrc (Timing Object Implementation)
    https://github.com/webtiming/timingsrc

    • Last updated: 7 June 2017
    • JS implementation of the Timing Object specification. Mainly developed by one person, with a current focus on media synchronisation (same media source). The synchronisation is very reliable and deals with a number of edge cases, but there are no reliable test cases for sequencing and time offsets (apart from a text-based proof of concept).
    • See also:

Comparison of timing / event concepts and implementations

As there is no native implementation for timebased events, all JS implementations rely on window.requestAnimationFrame, window.setInterval or the HTML5 media element > timeupdate Event (only possible if media element present) to regularly call updater / checker methods:

window.requestAnimationFrame()

window.setInterval()

The same observation was made in

Based on the interval (or animation frame) frequency, the updater / checker methods are used to compare the current time with the start / end timings of synchronised elements.

The requestAnimationFrame() method provides high performance, but the frequency can't be controlled (unless additionally using a window.setTimeout() call to delay its execution). Using the setInterval() method provides two advantages: 1) the frequency in which the updater / checker methods are called can be set to a fixed number, 2) several intervals of varying frequencies can be used to deal with low and high priority tasks (while high priority tasks are executed more often).