Dynamic Media Fragment
This is a proposal for dynamic spatial media fragments, as well as arbitrary shaped spatial media fragments, for highlighting moving areas in videos.
It adopts the principle of not reinventing the wheel, so since we are talking about graphical shapes and trajectories, SVG is a natural target.
The live demo exhibits different uses of the implementation.
The shape syntax is "simply" a SVG <path> definition, which brings interesting features :
- it is standardized
- it is compact and expressive
- it make simple things relatively simple (freehand shapes), while allowing more complex things (bezier curves, etc)
- there is software/libraries (such as d3.js) for editing it and generating it
- there is software for reading it, and most especially web browsers
- it can be styled through standard CSS syntax
The path definition is passed as the
shape= parameter. The
current implement considers a viewbox with the same dimensions as the
video widget, through this should be further specified and refined.
For example, the URL
video.mp4#t=10,50&shape=M0 0l0 50 50 50 50 0z
defines a 50pixel-wide square, in the top left corner of the video.
The current implementation defines the shape id according to the video
id. Given a video with the id
v1, the defined shape will have the id
v1_overlay_shape. This allows to style the element through standard
CSS, by specifying fill and stroke colors, patterns, etc (see
[main.css]). If no video id is specified, it will be automatically
This could be defined through a CSS pseudo-element such as
v1::overlay_shape, though it actually is an element, so it
probably does not match the specification.
The dynamic syntax also uses SVG paths for its definition, and makes the defined shape follow the given path during the duration of the fragment. The current implementation uses the browser internal SVG support for parsing and interpolating paths.
For example, the URL
video.mp4#t=10,50&shape=M0 0l0 50 50 50 50 0z&trajectory=M0 0L200 200
will move the aforementioned square in a diagonal line.
In order to facilitate debugging, you can add the
developement parameter, which will then display the trajectory in
addition to the moving shape.
If it is deemed desirable to have more common shapes available in SVG
such as circles and rectangles, the following extension is proposed
(and implemented): the shape element can optionnaly start with a shape
type specifier (one of
by a colon and by a type-specific list of data points.
For path (which is the default if no shape type is specified), a SVG path definition.
For rect, 4 numbers
For circle, 3 numbers
cx (center y),
cy (center y) and
For ellipse, 4 numbers
cx (center y),
cy (center y),
(horizontal radius) and
ry (vertical radius)
This proposal makes some simplifying assumptions, for which an appropriate trade-off has to be found.
A mediafragment does not define more than one shape. If multiple shapes must be defined, then multiple fragments should be used. Or you can define multiple shapes in a single SVG path (see demo page). Another idea could be to define the notion of MediaFragmentGroup, which would group multiplefragments, so that we can address discontinuous temporal or spatial ranges.
The shape itself does not change over its trajectory. This will not work for instance in case of a face tracking algorithm that follows a face in a shot, which could vary in scale and orientation. A possible way to solve this would be to extend the trajectory path syntax by adding a new "Scale" command, which would specify an object scale change, e.g.
trajectory=M9,323C147,8 E0.5 C270,78 E1.0 C270,78(using
Eas command abbreviation, since
Sis already used). The SVG standard defines an additional <animateTransform> element for this, but in XML syntax.
Similarly, the temporal specification is not flexible (it is a linear interpolation of the trajectory of the fragment duration in the current implementation). Temporal keypoints could be specified again by extending the path syntax, e.g.
t=0-10&trajectory=M9,323C147,8 K4 C270,78 K8 C270,78(using
Keypointabbreviation, since SVG defines the notion of Keypoints but with no associated compact syntax) would specify durations associated to the current trajectory command.
Due to the flexibility of the SVG path definition, the mediafragments will be more complex to interpret and query in applications such as the SPARQL-MM proposal.
For complex SVG paths, that are too cumbersome to be put in a single URL, a simple syntax extension could allow to reference external SVG files, e.g.
shapesrc=filename.svg. Note that this would make the dereferencing more complex and less robust (since the referenced file could be unavailable).
The dynamic path support has been implemented using the SVG API method
getPointAtLength, using a linear interpolation between the temporal
position and the position on the trajectory path. This provides a
compact representation, but does not allow to specify specific keypoints.
I have made some tests with standard SVG animation features (<animate> and <animateMotion>) but it proved impractical for 2 reasons :
browser support is variable, even though CanIUse states that there is good support
video synchronisation should be achieved by SMIL timing events (begin/end), but I did not find any way to make it handle seeks inside the fragment (i.e. setting the position to something other than the begin).
If anyone has experience with combining SVG/SMIL animation events with HTML5 video, feel free to comment.
Some more ideas that could be ironed out, time permitting:
using the shape as a clipping mask. It would involve defining it as a <clipPath> or <mask> element, and specifying the clip-path/mask CSS property for the video. Silvia Pfeiffer's HTML5 Video guide is a good starting point for this.
find a way to specify timing information. SVG animation uses keypoints, but in XML syntax. We could extend the trajectory path definition with timing information, but we would then to split the path into multiple segments before interpolating them.