Support EXT-X-GAP without jumping forward if audio is missing

[natez0r]

[REQUIRED] Use case description

We are investigating how to support “playing” through gaps in HLS segment data in ExoPlayer.
We have a use case where we do not want to insert discontinuity tags and skip over gaps in certain streams because we’re aligning them with other concurrently playing streams. We’ve noticed that if we note the duration of the segment to be larger than the amount of data in the actual segment, the player will stall due to the fact the media codec does not find frames to decode and the media clock does not continue to increment even though the forward buffer says we have data currently. The iOS AVFoundation player deals with this situation by displaying the last frame it decoded for the duration of the gap in data before picking up and displaying the first frame from the following segment at the appropriate PTS time.

Proposed solution

This could be addressed in the player by ensuring the player doesn't stall when no frame is decoded even though the loaders indicate we have sufficient forward buffer. I'm curious as to what the ExoPlayer team thinks about advancing the media clock in this scenario.

OR

This could potentially be fit for usage of the EXT-X-GAP tag. See draft spec, although the draft spec does not specify the behavior when no rendition exists without the gap.

Alternatives considered

Just the two from the Proposal section

[natez0r]

just wanted to follow up on this to see if y'all had any questions/initial reactions to the proposal?

[AquilesCanta]

This seems like a good fit for EXT-X-GAP, if I understand correctly. There three scenarios that may occur:

Scenario 1: Video and audio are demuxed, and an audio segment is missing.
Scenario 2: Video and audio are demuxed, and a video segment is missing.
Scenario 3: Video and audio are multiplexed, and a segment is missing.

Current behavior:
Scenario 1 and 3: An audio renderer discontinuity will cause playback to jump forward.
Scenario 2: The last video frame before the gap will remain rendered until the first video frame after the gap.

If I understand correctly, you would like in scenarios 1 and 3 the same behavior as scenario 2 currently has. There are two ways I can think of solving this: One is adding silence samples from the source and handling this properly in the renderer. Being the latter the one I think it's the proper way. That fix would be in the renderer.

@natez0r please confirm whether my interpretation is correct.

[natez0r]

I haven't tested to see the second scenario as I didn't have a demuxed stream, but that is the behavior I was hoping for.

Just to confirm: if you had a stream that did not have an audio track at all and we hit a gap, we'd still expect Scenario 2, correct?

One other thing to note is that the player behavior seemed to depend on the length of the gap in my testing. I think it may be different for demuxed streams, but if my forward buffer duration was shorter than the amount of time in the gap, the player would stall permanently

[AquilesCanta]

Just to confirm: if you had a stream that did not have an audio track at all and we hit a gap, we'd still expect Scenario 2, correct?

I think that is the current behavior, yes. That is: the last video frame before the gap will be on screen while "playing" the gap.

[natez0r]

Perfect, that's what we'd hope for.

I guess my biggest question going forward is whether or not this would be high priority enough for you to want to address it in house or should our team prepare a PR to address this and we can iterate with you on it?

[stevemayhew]

@AquilesCanta thanks for looking over this. I have a real world use case for this too.

Just to confirm: if you had a stream that did not have an audio track at all and we hit a gap, we'd still expect Scenario 2, correct?

I think that is the current behavior, yes. That is: the last video frame before the gap will be on screen while "playing" the gap.

For Scenario 2, I expect audio would continune playing and video would be frozen, much like a track switch to an audio only track on very low bandwidth (if we supported this). YouTube has behavior like this.

Here's another use case to consider (we are wrestling with this now). A channel has alternate audio or Spanish language, but only for a portion of the content (might drop out for ads or for some program content). Here we would like an event for the EXT-X-GAP so track selection could blacklist and switch to another audio rendition, as the spec "encourages":

The client SHOULD NOT attempt to load Media Segments that have been
marked with an EXT-X-GAP tag. Instead, clients are encouraged to
look for another Variant Stream of the same Rendition which does not
have the same gap, and play that instead

Or simply play silence. Note this might be tricky for tunneled playback, might need to disable audio renderer to allow media clock source from other than audio.

We are interested in helping out with this and can provide sample streams.

[AquilesCanta]

Thanks @stevemayhew for the input! We have considered multiple approaches in the past for this, although admittedly non of them was necessarily a simple approach. For the spec's suggestion, we considered using the already-built-in block-list mechanism in ExoPlayer. That is, when the HlsChunkSource tries to load an EXT-X-GAP, it will throw a special load exception which can in turn cause the block listing of the variant (like an HTTP 404 would do) and therefor we would load a segment from a different variant. It can get tricky when there are many gaps which are aligned, and eventually you are forced to just skip the gap (like we do now) if there's a section of the media covered by gaps across all variants.

I'll assign this to @christosts who's now looking into HLS. We'll reach back to look into possible steps forward. Your assistance with this is super valuable!