Image-based trickplay tracks

[technogeek00]

Use Case Description

From @nicoweilelemental

For player resources optimization reasons, the use of a video track as a trickplay artefact is not always possible, and a lot of player providers recommend the use of image thumbnails tracks instead of special low framerate video tracks. DASHIF IOP section 6.2.6 covers this use case but there is equivalent support in the HLS RFC. There is the Image media playlists HLS extension proposal from Roku/Disney/Warner here https://developer.roku.com/docs/developer-program/media-playback/trick-mode/hls-and-dash.md but its relevance/adoption is limited by the fact that it's not part of the RFC. Same logic here: even if not supported by Apple players which don't need it as they can leverage iframe-only tracks, it would be super useful for the rest of the ecosystem to get this officially part of the HLS RFC.

Working Notes

• Previously in DASH-HLS Interop: Image-based subtitles and trickplay tracks #2 but separated out to capture submissions independently
• Not something actively included in HLS RFC
• Per @mcpv comment the IETF HLS Extension is captured here: https://github.com/image-media-playlist/spec
• IETF hls-interest list good place for discussion, previous IETF thread that discussed: https://mailarchive.ietf.org/arch/msg/hls-interest/FCfPmXIU6YApjrH7UltpDVSezTA/
• Per @nicoweilelemental, his organization has now created a reference stream for both HLS and DASH

Open Questions

• With Apple's new proposal for MJPEG tracks how do we evaluate interoperability?

Prioritization Questionaire

Updated post Apple proposal

Question	Answer
Does the feature relate to an industry streaming use-case?	Yes, Thumbnails / Trickplay
- What is the commonality of this case?	Very Common
- Is this an established or emerging practice?	Established
Does this feature have related mechanisms in both DASH and HLS?	Yes
- What is the maturity of support in both specifications?	DASH - Mature, HLS - Immature
- What is the maturity of implementation support for both specifications?	DASH - Mature, HLS - Immature
- Are there known interoperability issues between specs?	Yes, HLS proposal is net new with no existing content
- Has anyone implemented this interoperably?	Yes, with non-spec aligned approach
- Are there features missing in a specification with open proposals for it?	Yes
Has the industry defined defacto mechanisms not present in both DASH and/or HLS?	Yes
- Why was functionality defined outside of the main specifications?	Industry desire
- Has the functionality been standardized elsewhere (DASH-IF, CTA, SVA)?	No
- Is the functionality proprietary or openly developed?	Proprietary, but open
- Could the functionality be incorporated into specifications with evangelism?	No

Original prior to Apple proposal

Question	Answer
Does the feature relate to an industry streaming use-case?	Yes, Thumbnails / Trickplay
- What is the commonality of this case?	Very Common
- Is this an established or emerging practice?	Established
Does this feature have related mechanisms in both DASH and HLS?	No
- What is the maturity of support in both specifications?	DASH - Mature, HLS - Missing
- What is the maturity of implementation support for both specifications?	DASH - Mature, HLS - N/A
- Are there known interoperability issues between specs?	Yes, HLS does not support
- Has anyone implemented this interoperably?	Yes
- Are there features missing in a specification with open proposals for it?	Yes
Has the industry defined defacto mechanisms not present in both DASH and/or HLS?	Yes
- Why was functionality defined outside of the main specifications?	Industry desire
- Has the functionality been standardized elsewhere (DASH-IF, CTA, SVA)?	No
- Is the functionality proprietary or openly developed?	Proprietary, but open
- Could the functionality be incorporated into specifications with evangelism?	Yes

[technogeek00]

Transfer of @nicoweilelemental comment:

Image-based trickplay tracks
HLS image-based trickplay track was already a fairly common feature for VOD contents targeting the Roku platform. Now it's spreading in the live domain too, as the spec extension is simple and efficient, and there is a growing demand for it.

Here is an AWS Elemental stream showing the implementation of the Image Media Playlist spec for the live use case: https://d345o62q7f2dwl.cloudfront.net/referencestream/main.m3u8

It's fairly easy to configure with MediaLive + MediaStore, or MediaLive + S3: https://docs.aws.amazon.com/medialive/latest/ug/trick-play-roku.html

The implementation work is happening on MediaPackage too, as we speak, and in the player ecosystem. That would be really great to capitalize on the Roku/Disney/Warner spec, which is the de factor industry standard now.

[technogeek00]

Additional note Disney Streaming's entire catalog will be aligned around this format.

[technogeek00]

12/15/2021 Working Notes

HLS Notes

• Extension Proposal to HLS RFC: Image Media Playlist
• Image based tracks
• Individual images can have tile thumbnails
  • Declared track resolution is size of individual thumbnails in tiled images
  • Each image in tile has standalone duration which is division of total
  • Reading of tiles is left to right, top to bottom

DASH Notes

• DASH-IF v4.3 6.2.6
• Image (jpg or png) based track with $Number$ based addressing
• Tiling is assumed to be a standard process component
• Resolution of the track is the full image, not an individual thumbnail
• Individual thumbnail resolutions are computed by sub-dividing the declared resolution by the tiling factors
• Each image in a tile has a start and duration computed by sub-dividing declared duration values and summing previous entries
• Reading of tiles is left to right, top to bottom

Interoperability

• Highly interoperable approaches, all information appears present!
• Restriction of naming to a derivable number based approach for sequence of frames
• Presentation time offsetting in DASH may translate to usage of EXT-X-GAP in HLS and vice versa

Open Questions

• Confirm: There is no restriction to JPG only in HLS proposal?

[nicoweilelemental]

On 2/23, Apple announced upcoming iOS support for M-JPEG (ISO/IEC 23008-12 "High efficiency coding and media delivery in heterogeneous environments —Part 12: Image File Format") encapsulated in fMP4 (CMAF?) containers. Signaling example:
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=331000,CODECS="mjpg",URI="iframe_index.m3u8"

M-JPEG prevents tiling but allows encryption. To be interoperable, it would need to be included in the DASH spec and supported by DASH players.

[technogeek00]

Pulling back into research phase due to the Apple M-JPEG announcement, will discuss in the call tomorrow.

[technogeek00]

Raised for broader discussion in CTA Spring Forum