Support tunneling in ExoPlayer

[chateaudif]

Hi there.
Is there any plans on adding tunneling support to ExoPlayer? Several Android devices (especially for Android TV) contain video decoders with tunneling support but it doesn't look like ExoPlayer attempts to take advantage of them yet. Some 4K streams are almost impossible to play properly without video tunneling.

thks,
L

[andrewlewis]

We looked into this a while back but found little performance difference between tunneling and normal playback on devices we tested. I think the primary claimed benefits at the time were around improved audio/video synchronization and allowing OEMs to customize decoding and rendering. We also found some device-specific bugs with stability and seeking, though these are likely fixed now.

Could you give some examples of devices where this feature is needed to play 4k streams? If there is sufficient demand may be able to prioritize it. Thanks!

[wendong-li]

@ojw28

Is the support for this feature under progress? I can see that the following recent changes in dev-v2:

e125ed7 Report tunneling support via RendererCapabilities API
cd42434 Plumbing to enable tunneling on AudioTrack from renderers

If so, is there a plan on when to complete the initial support?

Actually, we are also adding this feature internally in our organization as we thought the official support might not meet our timeline. Basically we require this feature by end of March this year.

[ojw28]

Yes, this is being worked on. As of the most recent commit MediaCodecVideoRenderer and both audio renderers are capable of tunneling.

To complete support we need to (a) check whether both audio + video renderers provide support for tunneling for the media being played, (b) generate an audio session ID for tunneling if they do (there's a static method in C to do this), and (c) propagate it to onEnabled of the renderers (there are currently TODOs in these classes to allow the ID to be propagated). There's some nuance involved in having to potentially disable and re-enable the renderers if you're in a playlist and they're toggling between having tunneling disabled/enabled, which we'll need to figure out.

We hope to complete this work soon. We don't give exact time estimates. Note that if you want to add the feature internally in the meantime then you'd be best off building on the changes so far, since they do all of what's required aside from the remaining piece described above.

[wendong-li]

@ojw28

Thanks for sharing the information. I rebased my changes on the latest dev-v2.

I think my understanding on the changes needed is aligned with yours. Since video tunneling needs to work together with audio hw av sync, the decision to enable video tunneling needs to be done in a place where both renderers' information can be accessed. After the decision is made, one thing left is to signal the enabling of tunneling in both renderers. As you pointed out, the multiple period use case makes the signal more difficult.

Our current change is here:
wendong-li@a907281

It's complete, but I couldn't test it yet as we are still waiting for the device with video tunneling support.

My guess is it's not same as what you have in mind on how to implement it. But in case you think it makes sense, you can apply it.

[ojw28]

My current plan is to have a track selector make the decision (since it knows about the capabilities of each renderer). The selector effectively becomes more of a binder that decides (a) which tracks to select, and (b) what the corresponding renderer configuration should be when using the selection. I'm not completely sure it's the best approach, as opposed to something directly in ExoPlayerImplInternal, but we're weighing it up internally and hope to have some news fairly soon.

[wendong-li]

@ojw28, thanks for the changes and sharing the information.

The current approach i.e. using track selector, seems to be more efficient to me. Looking forward to the final design.

Seems this feature is almost complete, except the following change that I'm aware of:
- Skip the output buffer processing in video renderer if tunneling is on

[ojw28]

I'm not sure any special logic is required for skipping output buffer processing. In any case, the solution appears to work fine end-to-end as it is.

[wendong-li]

I was referencing the CTS test case in MediaCodecTunneledPlayer.java for the usage of this feature. It suggests skipping the handling of video output buffer when tunneling is enabled.

Just checked the implementation of the native library, looks like when tunneling is on, empty buffer array would be returned from MediaCodec.getOutputBuffers() and dequeueOutputBuffer() would return TRY_AGAIN. That's probably why the current MediaCodecRenderer works without changes.