Render video for further processing

[b0g8]

Issue description

Is it possible to use the new Clock in such a way that rendering is not related to realtime? I'd like to apply some further processing to the video frames, and keep them all. Once the processing is done for one frame, advance the Clock so that the next frame is rendered. If possible, could you advise me about what could go wrong, or where to pay attention?

Reproduction steps

NA

Version of ExoPlayer being used

2.7.0

Device(s) and version(s) of Android being used

any

[botaydotcom]

I think the Clock interface is mostly for facilitating testing, and in fact some of our tests use custom clock implementations that are advanced manually. It was not designed for your use case, but it might work.

However. one problem I can think of for your approach is that in many places in ExoPlayer, we are still using SystemClock, and have not migrated to use the new interface yet. This is something that can be changed, but we don't have any plan to clean up this part anytime soon. Therefore, if you use your own Clock implementation, it may go out of sync with these components.
Some other things you may need to pay attention to:

• If you are working on a custom Renderer and extending from MediaCodecVideoRenderer, be aware that MediaCodecVideoRenderer will drop frames if they are late using a real time clock (SystemClock).
• If you play any Audio stream at the same time, the AudioTrack from system will use real time. Therefore, a manual clock for Video stream will make Audio and Video out of sync quite soon.
• Things like live streams (maybe not for your use case), will depend on real-time clock to work.

[andrewlewis]

One additional suggestion related to those points: if you are only rendering video you may be able to do this kind of batch processing easily by subclassing MediaCodecVideoRenderer, returning this from getMediaClock, and implementing MediaClock with getPositionUs returning the last rendered frame time (plus any additional offset required to make the next frame not too early) based on the presentation time passed to onProcessedOutputBuffer. (You can prevent frames from being dropped by overriding shouldDropOutputBuffer and shouldDropBuffersToKeyframe to return false, though frames shouldn't be late in this setup as you are advancing the player's position yourself.)

[b0g8]

Audio is rendered too, but to a file, not to the AudioTrack. Should the getMediaClock for MediaCodecAudioRenderer just return false? Will audio sync to video renderer time?

[andrewlewis]

Yes. You can only have one media clock at a time. The audio renderer doesn't do any synchronization when it processes output because it's providing the position.

Depending on your requirements, you may want to limit how far your audio renderer can read ahead of the video renderer (or fall behind, I guess, though this seems unlikely to happen in practice). I think they should be kept roughly synchronized by upstream buffering control, which is uses the player position coming from your video renderer. You can just return false from MediaCodecRenderer.processOutputBuffer whenever you don't want to handle the provided buffer yet.

[b0g8]

Audio data is processed as well. It may take longer than processing the video data. In order to limit how far the audio renderer can fall behind the video renderer, should the video renderer get the position of the audio renderer, and decide whether to advance or not?

[andrewlewis]

That sounds sensible and can't do any harm as long as you make sure the renderers can't get stuck not consuming any data. (Just to reiterate, the player can only have one enabled renderer that provides a media clock at a time. So you can't have both renderers provide positions in the "normal" way. I guess you'll just add a check at the start of processOutputBuffer for each renderer: one can use the player position and the other will directly check the last processed output buffer time of the other renderer.)

[AquilesCanta]

@b0g8 can this be closed?

[b0g8]

Very valuable information here, at least for me. Thanks.