Count droppedFrames in tunneling mode

[Cizor]

Searched documentation and issues

1. Exoplayer Issues
2. https://medium.com/google-exoplayer/tunneled-video-playback-in-exoplayer-84f084a8094d

[REQUIRED] Question

How to count dropped frames count when tunneling mode is enabled?
For non tunneling mode I am using decoderCounters.droppedBufferCount but code in MediaCodecVideoRenderer doesn't seem to handle it for tunneling mode. Please advise.

Link to test content

Content used from Demo app

[Cizor]

I am not very sure about it but since decoding of video is not in exoplayer code, it cannot determine if a frame is late, very late or should be dropped when tunneling mode is enabled

[krocard]

The tunneling decoder reports rendered frames with the callback MediaCodec.OnFrameRenderedListener.onFrameRendered which calls onProcessedTunneledBuffer which updates renderedOutputBufferCount but not the drop counter.
onProcessedTunneledBuffer could try to guess if a frame was not rendered by comparing the received timestamps to the expected ones.

As the tunneling decoders are not required to call onFrameRendered for every frame (and I have not tested if they do either). So it's not sure if the resulting metrics will be reliable.

Unfortunately I will not access to a tunneling setup until the London office reopens (COVID-19 confinement).

[krocard]

@Cizor, changxiangzhong in #7544 has done some testing regarding the tunneling onFrameRendered callback and it seems that the callback is reliable enough to detect frame drops, at least for their device.

I would encourage you to try the same methodology if possible to validate that it successfully detects drop frames for your devices.

[changxiangzhong]

@krocard I've found in the DebugMediaCodecVideoRenderer::onProcessedOutputBuffer() in dev-v2 branch, there are some logic regarding frame drop detecting. Source code is as following. I think we can reuse and modify it to detect frame drops.

    @Override
    protected void onProcessedOutputBuffer(long presentationTimeUs) {
      super.onProcessedOutputBuffer(presentationTimeUs);
      bufferCount++;
      long expectedTimestampUs = dequeueTimestamp();
      if (expectedTimestampUs != presentationTimeUs) {
        throw new IllegalStateException("Expected to dequeue video buffer with presentation "
            + "timestamp: " + expectedTimestampUs + ". Instead got: " + presentationTimeUs
            + " (Processed buffers since last flush: " + bufferCount + ").");
      }
    }

Essentially, this DebugMediaCodecVideoRenderer maintains a queue of PTS. After a packet has been pushed into the InputBuffer, its PTS would be put into the queue and sorted. So this queue contains all the PTS of frames that has been pushed to the codec but has not yet output to the Surface. Whenever there's a new output, it would compare the PTS and see if there's a frame drop.

[krocard]

I agree that this should be an effective approach to detect missing frame.