EXT-X-GAP

[mtoczko]

What version of Hls.js are you using?

0.14.7

What browser and OS are you using?

Chrome 84

Test stream:

gap: video segment 1.ts and 5.ts, audio segment 1.ts and 5.ts
https://mtoczko.github.io/hls-test-streams/test-gap-audio-video/playlist.m3u8

gap: video segment 1.ts and 5.ts
https://mtoczko.github.io/hls-test-streams/test-gap-video/playlist.m3u8

gap: audio segment 1.ts and 5.ts
https://mtoczko.github.io/hls-test-streams/test-gap-audio/playlist.m3u8

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client on https://hls-js.netlify.com/demo and not just on my page

• The issue occurs in the latest client on https://hls-js-dev.netlify.com/demo and not just on my page

• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Expected behavior

The client should not attempt to load segment.
Fill audio gaps with silence. (Safari)
Fill video gaps with black frame. (Safari)

[robwalch]

Thanks for the examples @mtoczko, and for creating this issue to point out the gap in our implementation.

Looking through the latest spec I see LL-HLS Partial Segments can also be flagged with a GAP attribute, so that's worth noting here since it looks like EXT-X-GAP was added at the same time as LL-HLS tags.

6.3.3. Playing the Media Playlist File
...
The client SHOULD NOT attempt to load Media Segments that have been marked with an EXT-X-GAP tag, or to load Partial Segments with a GAP=YES attribute.

[robwalch]

#2865 Parses GAP, although functionally the result is the same as before; All unknown tags starting with # in the playlist get added to the fragment tagList.

Filling with silence is doable. We don't have any code that can generate black frames. At best we can copy or extend the duration of existing frames.

[mtoczko]

@robwalch Safari fills the gap with the last video frame. If the gap is in the first place, it takes the first frame of the next segment and extends the duration.

[robwalch]

Thanks for clarifying @mtoczko. That makes sense, and would be a great feature, not just for GAP segments, for for actual start gaps that result in stalls and require seek nudging.

[robwalch]

Not a Contribution

From the HLS spec:

6.3.3. Playing the Media Playlist File

...

The client SHOULD NOT attempt to load Media Segments that have been
marked with an EXT-X-GAP tag, or to load Partial Segments with a
GAP=YES attribute. Instead, clients are encouraged to look for
another Variant Stream of the same Rendition which does not have the
same gap, and play that instead.

So after avoiding segments or parts marked with a GAP, and before filling with silence or extending frames, we should attempt to find media in other renditions.

[ok-ikaros]

hey @robwalch I see that this feature is in the list of top priorities for the next release - do you guys know what the status on this?

[robwalch]

Hi @lelandhwu,

The Milestone is set to 1.4.0 which is the next planned release: HLS.js Roadmap.

The work hasn't started yet. I need to wrap up #5191 first. The plan is to skip loading of segments with a GAP tag, and handle them the same way a FRAG_PARSING_ERROR would be handled.

[mtoczko]

Hi @robwalch
I am still analyzing the algorithm of operation. But I have a few questions:

What is the source of the value of this constant? RENDITION_PENALTY_DURATION_MS = 300000
I have noticed that there is no switch to the backup stream for the audio track, is this intentional?
In the gap of an audio track, can we fill the stream with silence? Similar to what happens in the native Safari player?

[robwalch]

What is the source of the value of this constant? RENDITION_PENALTY_DURATION_MS = 300000

It's an arbitrary value. We could make it configurable in the future. It exists so that clients to flip back and forth between redundant streams (or hosts). If a set errors enough to be put in the "penalty box" it will not be switched back to for five minutes.

I have noticed that there is no switch to the backup stream for the audio track, is this intentional?

No? If an audio or subtitle track errors the whole rendition (the variants using that group) error, and the player switches to the backup (or should switch to a variant with a different group).

[robwalch]

v1.4.0 will include support added in #5257. This issue can be used to track additional enhancements and feedback.

[mtoczko]

Hi @robwalch
I noticed one undesirable action: gap-controller.ts:264 [warn] > skipping hole, adjusting currentTime from 105.275674 to 170.05533300000002

To reproduce the issue, you need to watch the entire video and seek to 105.
"backBufferLength": 90
08b938f3.hls-js-dev.pages.dev-1680185898252.log

[robwalch]

Hi @robwalch I noticed one undesirable action: gap-controller.ts:264 [warn] > skipping hole, adjusting currentTime from 105.275674 to 170.05533300000002

Filed #5360

[mtoczko]

Hi @robwalch

No? If an audio or subtitle track errors the whole rendition (the variants using that group) error, and the player switches to the backup (or should switch to a variant with a different group).

Maybe you have an idea why it plays the video normally once and switches the group, but when played again it skips the gap.

[log] > [audio-stream-controller]: Buffered audio sn: 100 of track 0 (frag:[200.000-202.005] > buffer:[50.005-142.016][154.005-156.011][168.000-186.005][188.011-202.005])
base-stream-controller.ts:1695 [log] > [audio-stream-controller]: PARSED->IDLE
base-stream-controller.ts:715 [log] > [audio-stream-controller]: Loading fragment 101 cc: 0 of [0-133] track: 0, target: 202.005
base-stream-controller.ts:1695 [log] > [audio-stream-controller]: IDLE->FRAG_LOADING
gap-controller.ts:304 [warn] > skipping hole, adjusting currentTime from 141.952299 to 154.05533300000002

80d1619f.hls-js-4zn.pages.dev-1680353100256.log

[robwalch]

Maybe you have an idea why it plays the video normally once and switches the group, but when played again it skips the gap.

I've just made some improvements to this in #5366 that allow for more variant and redundant rendition failover (back to renditions that already failed).

The thing is with redundant failover we don't maintain the level details for alternates (only roughly in the error handler now for this issue). The failover is not something we want to repeat when both renditions have errors, so there is some basic balancing and penalization of renditions selection based on error count where we do not switch back to alternates if they have errored more the the current selection and errored recently.

If you add PATHWAY-ID to the redundant variants then the player would use Pathway Switching rather than redundant failover to organize levels and handle the failover switching. This will change behavior such that each variant or level is a different object rather than one level with redundant urls and attributes grouped in the level object. The error penalty may be harsher (not allowing for failover back to pathways that errored recently) but because the data is there, fragment gaps could be compared across all options. In the future, redundant levels may be refactored discrete objects with assigned pathways, but for now, that implementation (level.urls and level.urlId) remains the same.