The last image in the thumbnails is not always correct

[avelad]

Have you read the FAQ and checked for duplicate open issues? Yes

What version of Shaka Player are you using? 3.1.1

Can you reproduce the issue with our latest release version? Yes

Can you reproduce the issue with the latest code from master? Yes

Are you using the demo app or your own custom app? Both

If custom app, can you reproduce the issue using our demo app? Yes

What browser and OS are you using?
Chrome 91 macoS 11.4

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

What are the manifest and license server URIs?

I'll send a email

What configuration are you using? What is the output of player.getConfiguration()?

Default

What did you do?

Load a stream with thumbnails as matrix, ah get the last position with player.getThumbnails

What did you expect to happen?
The thumbnail position returned is correct

What actually happened?

The thumbnail position returned is not correct

Example:
Stream duration: 1460
Number of images: 2

First image
Start: 0
End: 1000

Second image
Start: 1000
End: 1460

The stream duration is 1460, but the thumbnail when using a matrix, the duration of the image track may exceed the duration of the audio/video to be able to fill the matrix correctly. In this case the second image ends in 2000

This is a problem that only occurs in DASH using a matrix.

[avelad]

@paulJordaan do you have the same issue?

[paulJordaan]

Personally, I have only tested with the following testing assets and I think they had full image matrixes present so I did not run into this edge case.

• https://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel-tiled-thumbnails-timeline.ism/.mpd
• https://dash.akamaized.net/akamai/bbb_30fps/bbb_with_multiple_tiled_thumbnails.mpd

If you could share the manifest you are using in this case I can assist with further troubleshooting.

[avelad]

@paulJordaan I can share privately. Do you have an email to send it to?

[paulJordaan]

Great, you can send to pdjordaan@gmail.com

[avelad]

Done!

[paulJordaan]

@avelad I've been looking at this issue and hoping you can help me with a question I have.
Do you know if it is possible to access the duration of the Imagetracks as contained in the manifest?

<AdaptationSet id="4" group="5" mimeType="image/jpeg" contentType="image">
      <SegmentTemplate timescale="1000" duration="1000000" media="DLVY1581840855002521-$RepresentationID$-scale=1000-$Number$.jpg" />
      <Representation id="thmb_2000_THMB=6" bandwidth="2000" width="1600" height="900">
        <EssentialProperty schemeIdUri="http://dashif.org/guidelines/thumbnail_tile" value="10x10" />
      </Representation>
    </AdaptationSet>

Having access to this value will allow us to calculate the duration of a single thumbnail window ie 10s and from there correctly calculate the value of total images available in a thumbnail matrix. ie

Duration for Representation = 1000s
Duration for a single thumbnail = TotalDuration / Number of thumbnails in matrix (rows x cols)
                                                     = 1000 / 100
                                                     = 10s
                                                     
When we have the duration of a single thumbnail we can calculate the real number of thumbnails available for a segment
Duration of the second image = end - start
                                                   = 1460 - 1000
                                                   = 460s
                                                   
460 / 10s = 46 thumbnails

Currently totalImages is calculated as cols*rows which gives us false values as the size of the grid does not directly relate to the duration of the segment

[avelad]

@joeyparrish can you help us with it?

[joeyparrish]

Here's what I'm seeing with the content @avelad sent:

1. The content duration (mediaPresentationDuration in the MPD) is 1460.
2. The image stream has duration 1000000, timescale 1000, meaning 1000s segments.
3. There are therefore only 2 segments, the second of which is full of blank space not used for thumbnails.
4. Calling player.getThumbnails(/* id / 6, / time */ 0) gives me the first segment, position 0,0.
5. Calling player.getThumbnails(/* id / 6, / time */ 1000) gives me the second segment, position 0,0.
6. Calling player.getThumbnails(/* id / 6, / time */ 1460) gives me null.

So we're getting the correct thumbnail, we just need to know how long to show it.

The duration of each segment is known to the manifest (in SegmentReference, 1000 in this case). The duration of each thumbnail in the segment is the segment duration / total images in the grid (including blank ones). We can compute this and expose it through the Thumbnail interface.

I've got a draft locally which I will try to get through code review today.

[joeyparrish]

Ah, now I see it's not quite as trivial as I thought. There's an edge case, which is that the segment duration of the last segment is clipped to the media duration. This results in a calculated thumb duration of 4.6 seconds instead of 10, since the last segment is clipped to 460 seconds to match the end of the presentation.

[joeyparrish]

This might get fixed today, or maybe tomorrow. Since the thumbnail feature is broken without this fix, I'll hold v3.2 for another day to make sure this is in.

[avelad]

It's really only broken when you don't use a 1x1 matrix. When I did the initial implementation, my streams used 1x1, and the Demo ones don't have this case.

Note: I agree with you that putting it in 3.2 would help me avoid workarounds.

Thanks @joeyparrish !

[joeyparrish]

I have a fix in review. Thanks!