WebVTT thumbnail/trickplay support

[JonWatson]

It is common for a WebVTT file to accompany a manifest which includes references to locations on a sprite sheet. These can be used for thumbnails in the seekbar (trickplay) like so

WEBVTT

00:00:00.000 --> 00:00:05.000
f08e80da-bf1d-4e3d-8899-f0f6155f6efa.jpg#xywh=0,0,120,67

00:00:05.000 --> 00:00:10.000
f08e80da-bf1d-4e3d-8899-f0f6155f6efa.jpg#xywh=120,0,120,67

--

Does Media3 provide any support in the parsing of this VTT file and the transformation into a small image given a playback position?

Thank you

[icbaker]

ExoPlayer does support thumbnail tracks in other contexts, e.g. when specified as contentType="image" and with schemeIdUri="http://dashif.org/guidelines/thumbnail_tile" in DASH. You can see an example test file here: https://github.com/androidx/media/blob/main/libraries/test_data/src/test/assets/media/dash/thumbnails/sample.mpd

The example WebVTT you've provided seems indistinguishable from a subtitle WebVTT file - so without any extra info I'm not sure how ExoPlayer could know how to interpret it differently.

Is there some wider context we could rely on (e.g. is this specified via a DASH or HLS manifest or similar)?

Could you share a link to a complete stream that you would expect to work so we can inspect it to understand it better? Please either share a link here or send it to android-media-github@google.com with the subject Issue #2706. Please also update this issue to indicate you’ve done this.

[efkann]

ExoPlayer does support thumbnail tracks in other contexts, e.g. when specified as contentType="image" and with schemeIdUri="http://dashif.org/guidelines/thumbnail_tile" in DASH. You can see an example test file here: https://github.com/androidx/media/blob/main/libraries/test_data/src/test/assets/media/dash/thumbnails/sample.mpd

The example WebVTT you've provided seems indistinguishable from a subtitle WebVTT file - so without any extra info I'm not sure how ExoPlayer could know how to interpret it differently.

Is there some wider context we could rely on (e.g. is this specified via a DASH or HLS manifest or similar)?

Could you share a link to a complete stream that you would expect to work so we can inspect it to understand it better? Please either share a link here or send it to android-media-github@google.com with the subject Issue #2706. Please also update this issue to indicate you’ve done this.

As far as I know, WebVTT thumbnails aren’t tied to HLS or DASH manifests. They’re typically delivered as a separate resource that accompanies the main stream, rather than being embedded in the manifest itself. It's surprisingly common as I’ve come across it on several major streaming platforms.

See:

https://developer.bitmovin.com/playback/docs/webvtt-based-thumbnails

As you said, the file itself looks identical to a subtitle WebVTT, so ExoPlayer can’t infer that it should be treated as thumbnails unless explicitly configured.

[icbaker]

They’re typically delivered as a separate resource that accompanies the main stream

OK thanks, that's useful context, so it's similar to 'sideloaded' subtitles where e.g. a WebVTT subtitle file is provided alongside a video file, and (as far as ExoPlayer is concerned) nothing ties them together until they're both added to the same MediaItem as the uri and subtitleConfiguration respectively.

A thumbnail seeking UI in ExoPlayer generally requires two players: One to play the video content, and one to show the thumbnails (so that you can scrub around in the thumbnail player while the video player keeps playing, or remains paused).

So it wouldn't make sense to add explicit 'thumbnail sideloading' support to MediaItem, since we wouldn't expect to need a single MediaItem that encapsulates both the video and thumbnail content, since they will be passed to different players.

Which leads me to think that one way to implement this would be to accept a WebVTT file in MediaItem.localConfiguration.uri (rather than SubtitleConfiguration), and handle that as an image track. This needs a bit more thought, we may want to also define a specific "thumbnail WebVTT" MIME type that can be set in MediaItem.localConfiguration.mimeType - so it makes the decision making totally clear.

I'm afraid we are unlikely to implement this support ourselves soon, but I'm happy to continue design discussions in this issue if someone wants to put together a PR (or PRs) for this.

[JonWatson]

@icbaker Thanks for your responses.

Given a web URL for the WebVTT, it's relatively simple for a developer to download the file, parse it into time -> sprite locations or bitmap mapping, store that internally and look up the needed trickplay bitmap on demand, all while cleaning up after ourselves afterwards (releasing the bitmap(s))

Indeed, I have implemented this with no issues since my original message.

Now that I know more about it, perhaps this is a feature that should always live outside of ExoPlayer's ecosystem. But given that ExoPlayer does internally already have WebVTT parsing built-in, I thought it might be something it could potentially support with something similar to what I've done. There are several details (such as the distinction that the position value used for the position -> bitmap function is not necessarily the position of the player but rather a separate, not-yet-seeked position of the seekbar) that might make tying this into ExoPlayer's APIs tricky.

I'll think about this more and if I come up with something clean perhaps I'll open a PR.

[google-oss-bot]

Hey @JonWatson. We need more information to resolve this issue but there hasn't been an update in 14 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!