Ability to switch audio track during playback

[IanDBird]

I'm attempting to add support to our video player that allows individual audio tracks to be selected during playback. Initially i'm focussing on my ExtractorRendererBuilder (with MP4s) but this really affects any media which can contain multiple different audio tracks.

First Issue

MediaCodecAudioTrackRenderer by default will select, and use, the first audio track provided by the SampleSource. It therefore seemed appropriate to extend this class and override the handlesMimeType used during onPrepare in order to try and "select" the audio stream that you want. A problem then arrises that "mimeType" is not a unique identifier within a video. For example, you could easily have an MP4 which has 2 AAC tracks, the main feature vs the directors commentary. In order to make it easier to identify individual tracks, it might be worth considering including the trackIndex into the method, for example:

protected boolean handlesMimeType(int trackIndex, String mimeType)

With this change in place, it should then be possible to create N individual audio track renderers (one for each known audio track), and to disable / enable them on demand.

Second Issue

There are two possible (related) issues with what I described above:

• Are there any overheads associated with having lots of separate instances of MediaCodecAudioTrackRenderer?
• What if you don't know up front how many audio tracks are available?

The audio tracks are available via the SampleSource but this is prepared after the MediaCodecAudioTrackRenderer's have been created. I therefore looked at what it would take to have a single MediaCodecAudioTrackRenderer that when we needed to "change" it (because we needed to select a new track index) we would attempt to re-use the original SampleSource but create a new instance of the audio track renderer. However, there doesn't appear to be any way to replace/substitute a new track renderer like this. As mentioned in my approach above, the handlesMimeType is called when being prepared. However, if you call the ExoPlayer.prepare a second time it will dispose of all the original renderers and eventually their SampleSource (that we're trying to re-use).

Is there something that i'm missing on how best to implement this?

[Ood-Tsen]

It seems only SS and Dash are support switch between multiple audio tracks.

Those have implement MultiTrackChunkSource.

[ojw28]

Correct. Also, when we implement this for HLS, we'll use MultiTrackChunkSource there as well.

I think we're less interested in the ExtractorSampleSource case in general, since muxing multiple tracks of the same type into a single stream is fundamentally not the best approach (DASH or SS is always going to be better if you want multiple tracks of the same type). If you were to add support, however, doing it at the ExtractorSampleSource layer would be more consistent with the model used elsewhere. You shouldn't need multiple renderers. It should be possible to switch track by (1) pause, (2) disable renderer, (3) reconfigure sample source to provide samples for a different track, (4) re-enable renderer [if applicable], (5) unpause [if applicable]. Like:

https://github.com/google/ExoPlayer/blob/master/demo/src/main/java/com/google/android/exoplayer/demo/player/DemoPlayer.java#L600

Step (3) is something you'd need to implement.

[IanDBird]

I completely understand and agree that there might be better formats out there which handle multiple tracks of the same type. However, since it's basically common practice to store multiple audio tracks within a single MP4/MKV/whatever, it does feel like a useful/important feature to support.

I've had a go at working through what you suggested, and still struggling a little. The renderer and selected track within the sample source are fairly tightly coupled, the MediaCodecTrackRenderer even stores the trackIndex as a local member. When the original renderer is "disabled" it will go ahead and disable the same associated track within the SampleSource. In order to ensure the MediaCodecTrackRender is initialised correctly when needing to select a different track, it needs to execute it's doPrepare again in order to allow someone to programmatically control the track to be selected (via handlesMimeType). But as I mentioned in the original description, even if this was called, handlesMimeType's interface isn't good enough to uniquely identify an actual stream.

If I start changing the ExoPlayer source to allow trackIndex to be modifiable, I guess I could do something like:

• Disable the MediaCodecAudioTrackRenderer, which disables the current audio track inside the SampleSource
• Set trackIndex on my MediaCodecAudioTrackRenderer instance to the new track i'd like to consume
• Enable the MediaCodecAudioTrackRenderer

I'm sorry, but is there any other way to do this cleanly? I'm trying to avoid deviating from the standard ExoPlayer source as much as possible (to ensure it's easier to pick up changes).

[IanDBird]

Just a quick update, but I have got things working using the following code:

A custom audio codec renderer:

public class SelectableMediaCodecAudioTrackRenderer extends MediaCodecAudioTrackRenderer
{
  public static final int MSG_SELECT_TRACK = 1;

  public SelectableMediaCodecAudioTrackRenderer(
      SampleSource source,
      DrmSessionManager drmSessionManager,
      boolean playClearSamplesWithoutKeys,
      Handler eventHandler,
      EventListener eventListener)
  {
    super(source, drmSessionManager, playClearSamplesWithoutKeys, eventHandler, eventListener);
  }

  @Override
  public void handleMessage(int what, Object msg) throws ExoPlaybackException
  {
    if (what == MSG_SELECT_TRACK)
      super.trackIndex = (Integer) msg;
  }
}

Code to switch the audio track:

      m_player.setPlayWhenReady(false);
      m_player.setRendererEnabled(TYPE_AUDIO, false);

      m_player.sendMessage(m_audioRenderer, SelectableMediaCodecAudioTrackRenderer.MSG_SELECT_TRACK, newStreamIndex);

      m_player.setRendererEnabled(TYPE_AUDIO, true);
      m_player.setPlayWhenReady(true);

The thing that's not possible without modifying the ExoPlayer source is the super.trackIndex = (Integer) msg;. Ideally, there would be a way for a subclass to programmatically select the desired track, as long as it was of the same type and also the renderer itself was disabled.

Would that change be considered?

[IanDBird]

This looks like a related issue: #168

[ojw28]

In my proposal there would be a single track exposed by the sample source, and the switching would be hidden behind it. The approach definitely works; if you look at how we do it for DASH/SS, it's following that approach.

Having said that, it seems things would be better if track switching were made more of a first class citizen of ExoPlayer. I think we should do the following, which should be good for what you need:

1. Add a onPrepared() method to ExoPlayer.Listener, to be invoked once preparation has been completed (you can infer the same thing from the existing onPlayerStateChanged, but it's not trivial to do so).
2. Add getTrackCount(), getTrackInfo(int index) and selectTrack(int index) (or similar - exact API TBC) to TrackRenderer.
3. Add getTrackCount(int rendererIndex), getTrackInfo(int rendererIndex, int trackIndex) and selectTrack(int renderer, int trackIndex) to ExoPlayer.

We'll need to figure out what an appropriate TrackInfo object is. It will need to contain more than the current TrackInfo object.

To implement multi-track selection in an app you'd then:

1. Wait for onPrepared, which indicates that track information is now available.
2. Query the track information and display it in the UI.
3. Call selectTrack(rendererIndex, trackIndex) to select tracks.

It may also make sense to merge setRendererEnabled into this new API. Disabling a renderer could be implemented by calling selectTrack(renderer, -1).

Thoughts?

[ojw28]

I have a version of this working, but it's pretty major surgery through the code-base. It'll likely be a few weeks until it's all merged.

[nahojkap]

Assume you are going to use 2 separate chunk sources for this use case? 1 for video and the MultiTrackChunkSource for the various audio tracks?

If so, there is a big risk the two will not correctly synchronize audio since the loading of playlists can (and will) happen at different times and under the right circumstances, the player will choose different start segments to start with.

For example:

Video playlist is loaded and contains sequence numbers 5,6,7,8,9,10 - playback will start with 8
Audio playlist is loaded and contains 6,7,8,9,10,11 - playback should start with 8 but code currently will pick 9.

So, there is a need to synchronize this somehow between the chunk sources. Perhaps you have already considered this but I thought I'd bring it up.

[ojw28]

This request is unrelated to HLS. Multi-audio in HLS is tracked by #73.

[nahojkap]

My bad - will the work you are doing here bring #73 closer to being implemented too? In any case, will re-post this there.

[ojw28]

It will provide plumbing for passing track information through common player components, so yes it will help.

[maxstone2012]

where download the source code that the MediaCodecAudioTrackRenderer Ability to switch audio track

[IanDBird]

Hey @ojw28, any updates on how this is progressing? If there is anything I can do to help out, please just let me know!

[ojw28]

We'll start merging some steps towards this soon. The first one will create a common base class for all TrackRenderer implementations that pull samples from a SampleSource, so as to avoid having to add multi-track propagation logic in multiple renderers separately. We'll then start merging code to actually propagate the tracks through the various components. Initially we'll be exposing fairly useless information about each track, although once we're at that point the task of populating the information properly can probably be parallelised to some extent.

[ShaoWeiguo]

BTW, when will it be done? We also need this feature.

[ojw28]

Status update:

• This is fixed as far as MP4 is concerned. If you try and play an MP4 with multiple tracks in the demo app, the Audio button will now list each track separately, and allow you to switch between them.
• For SmoothStreaming you'll see multiple audio tracks as before. For video, you'll now see both "auto" and each individual format. Previously it was not possible to switch to a fixed format, so this is new.
• For DASH, multiple audio tracks have temporarily disappeared. They'll be back in due course, along with the ability to select individual video formats (i.e. equivalent to the SmoothStreaming case). This work is currently blocked on another feature being delivered.
• HLS is the same as before, except that the video format exposed is currently wrong (it should probably be "auto"). We'll fix this.

[ojw28]

• We will also be adding multi-track support for MKV/WebM (probably tomorrow).

[dnutcracker]

Does this means it will also be supported to mute the video while it's playing? (by switching to a null/empty audio track)

[andreimarinescu]

@dnutcracker: you should be able to achieve this already by doing something like this in your ExoPlayer interface (equivalent of the DemoPlayer class of the demo app):
player.sendMessage(audioRenderer, MediaCodecAudioTrackRenderer.MSG_SET_VOLUME, 0f);

[ojw28]

This has been possible for a long time (the demo app has had this ability for a long time too). Using ExoPlayer's setRendererEnabled method is the best way to do it in the current release.

[dnutcracker]

Ok.Thanks.

On Fri, Aug 21, 2015 at 12:30 PM, ojw28 notifications@github.com wrote:

This has been possible for a long time (the demo app has had this ability
for a long time too). Using ExoPlayer's setRendererEnabled method is the
best way to do it in the current release.

—
Reply to this email directly or view it on GitHub
#514 (comment).

[ojw28]

• Multi-track is now fully working on the dev branch for SmoothStreaming, DASH, MP4, MKV and WebM.
• HLS multi-track is tracked separately (Change video track (Quality) on HLS stream #676, Implement multiple audio tracks support for HLS #73).