ExoPlayer drops last video frame of the segment on bitrate change

[shalin186]

Hello,

Based on my debug on ExoPlayer 2.6.0, I see that the current NAL unit (Type: Access Unit) related information is written to SampleMetadatQueue (commitSample() function) when the next NAL unit (Type: Access Unit) is found. So basically, when we find the start of Next NAL unit, we know that the previous NAL unit has ended and we commit metadata for that NAL unit. Which generally works fine.

But during that bit-rate change, I see that following happens in HLSMediaChunk.java:

    if (previousChunk != null) {
      shouldSpliceIn = previousChunk.hlsUrl != hlsUrl;
      previousExtractor = previousChunk.discontinuitySequenceNumber != discontinuitySequenceNumber
          || shouldSpliceIn ? null : previousChunk.extractor;
    } else {
      shouldSpliceIn = false;
    }
    Pair<Extractor, Boolean> extractorData = extractorFactory.createExtractor(previousExtractor,
        dataSpec.uri, trackFormat, muxedCaptionFormats, drmInitData, timestampAdjuster);

This means, we create new extractor when the bitrate (hlsUrl) changes. This new extarctor ends up creating new objects for H264Reader and Sample reader and hence the old state is lost. Now when the first NAL unit of the new chunk (at new bit-rate) is found, we are not able to commit Metadata for last NAL unit of previous chunk (at old bit-rate). And hence the last frame is never read/decoded/displayed.

This can be reproduced by any multi-bitrate HLS stream. Can some one in ExoPlayer team take a look at this and validate if this is actually an issue?

Shalin

[ojw28]

Interesting observation. Is it mandated anywhere in the HLS specification that a sample cannot be split across more than one segment, because I'm fairly confident we've seen streams that do this. If it's not mandated, I don't think we know that the last sample is complete in the case you describe, because it may be continued in the next segment of the bitrate we've moving away from. Hence it would be unsafe to output sample metadata for it, because this could lead to the decoder failing upon being provided with an incomplete sample.

To conclude, I think you're right, but I'm also not sure it's fixable. It seems like a fundamental problem with use of TS in HLS. If any of my observations or assumptions above are incorrect, and it can be fixed in a safe way, please let us know! It should be noted that using FMP4 streams, either in HLS or (better) DASH, solves the problem, because the FMP4 container makes it obvious where the end of each sample is.

[shalin186]

Hi ojw28,

Thanks for your detailed response. I now understand why it is done this way. I was not aware of HLS spec where it can allow sample to cross the segment boundary. In our case though, that's not going to happen since we support the hls streams created only by us (which won't have such situation). So in case, we know that the sample will not cross the segment bondary, how would you recommend I can fix this issue just for my individual case? I know you can't fix it in ExoPlayer repo but some pointers for my local fix would be appreciated.

Shalin

[ojw28]

If you control the server side then I'd suggest you upgrade to using FMP4 segments, and get all of the other benefits that brings at the same time. FMP4 segments have been supported in HLS for a while now.

For a local fix, you could try something like modifying TsExtractor to execute some flushing logic when read is about to return RESULT_END_OF_INPUT, where the flushing logic causes any pending samples to be output.