Support addition of tracks mid-stream

[lion4509]

[REQUIRED] Searched documentation and issues

I already have searched on this pages but not found any similar question.

[REQUIRED] Question

I am developing multicast player based on ExoPlayer version 2.10.1.
There are programs that are including DVS (Descriptive Video Service).
That means during playback an audio track could be added into the stream.
I thought onTrackChanged is called when the track is added but didn't.

[tonihei]

Can you please fill out the required sections for the content not playing template? Especially reproduction steps, example media, devices, etc. are helpful to understand and reproduce the problem.

[lion4509]

[REQUIRED] Content description
The content is multicast. It includes an audio track but another audio track is added during playback (DVS: Descriptive Video Service).

[REQUIRED] Link to test content
https://drive.google.com/open?id=1zy7-BlhgcVBdcI0Tlbo0FwC_Qo27kvO5
As I explained, extra audio track is added during playback at 4:45 but can not detect the audio track. I tested it also on ExoPlayer demo app and checked audio tracks using SELECT TRACKS button but there was only original audio track not extra one.

[REQUIRED] Version of ExoPlayer being used
2.10.1

[REQUIRED] Device(s) and version(s) of Android being used
AOSP Set-Top Box (Android 8.0 Oreo)

[REQUIRED] A full bug report captured from the device

[tonihei]

@AquilesCanta Can you look at the example and check if that's a valid use case? If so, we should probably mark as an enhancement.

[lion4509]

@AquilesCanta Do you have any update on this?

[lion4509]

@tonihei Can you assign anyone to resolve this issue? Seems like @AquilesCanta has been absent.

[AquilesCanta]

I'll look into this when I have some time. Thanks for your patience.

It's worth noting that ExoPlayer does not support the addition of tracks mid-stream, and it's unlikely that's changing any time soon. I'll check the content to make sure this is the case soon, though, but it's possible we won't be able to fix this in the short term.

[lion4509]

@AquilesCanta
I have tried to solve this issue on my side but not easy.
Have you checked the content and got any solution?
I know you are not sure it will be changed any time soon but please tell any advice.

[AquilesCanta]

I can confirm a track is being added mid-stream. As mentioned above, this is not something ExoPlayer currently supports. I'll use this issue to track support for addition of tracks mid-stream.

Solving this issue is not simple. One way I can think of solving this issue without changing the core representations in ExoPlayer is creating a new MediaSource that, when a track change is detected, signals the end of the current period and the start of a new period that includes the additional tracks. Notes:

• Both the old and new periods must share the ownership of the loader that owns the http connection.
• Will require changes in the extractor to signal the addition of a track.

[lion4509]

@AquilesCanta Thanks for your response.
I have queries about your suggestion.

If I could detect a track changes from the extractor, how the new MediaSource can signal the end of the current period and the start of a new period? Did you mean releasePeriod & createPeriod method? Please tell more specifically.
And worried about that the playback is stuttered(playing after stopped a moment) when additional track is added. What about you?

[AquilesCanta]

And worried about that the playback is stuttered(playing after stopped a moment) when additional track is added. What about you?

Not sure, not necessarily. You'd have to experiment.

If I could detect a track changes from the extractor, how the new MediaSource can signal the end of the current period and the start of a new period?

You can start by reading about Timeline, MediaSource and MediaPeriod (ideally, have a look at the code from the dev-branch). When MediaSource.prepare is called, the MediaSource will notify back (through onSourceInfoRefreshed) the shape of the media represented by a timeline, which is conformed of periods. The MediaPeriods are the objects in charge of loading the media of each of the periods in the media.

Did you mean releasePeriod & createPeriod method?

No. These are called by the player in accordance to what the player is currently playing and loading. You should have a look at the ProgressiveMediaSource and DashMediaSource implementations.

[AquilesCanta]

Note that we will eventually address this issue, and it's possible that a multi-period approach is not the best way to go, so keep in mind this approach may have flaws (like non-continuous playback). If you come up with other ideas, it's worth exploring.

[lion4509]

@AquilesCanta
I tried below codes but non-continuous playback happened (1~2 sec).
What about on your side? Any updates on this?

create MediaSource and called player.prepare()

mediaSources = new MediaSource[2];
for (int ix = 0; ix < mediaSources.length; ix++) {
    ProgressiveMediaSource source = new 
            ProgressiveMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
    source.setSectionUpdateListener(this);
    mediaSources[ix] = source;
}

dataSourceFactory.getTsStreamer().startStream(uri,getPlayerId());
player.prepare(new ConcatenatingMediaSource(mediaSources));
player.setPlayWhenReady(true);

updateSource() when dectected additional track

Timeline timeline = player.getCurrentTimeline();
if (!timeline.isEmpty() && !player.isPlayingAd()) {
    int windowIndex = player.getCurrentWindowIndex();
    int nextWindowIndex = player.getNextWindowIndex();
    
    if (windowIndex == 1) {
        nextWindowIndex = player.getPreviousWindowIndex();
    }
        
    if (nextWindowIndex != C.INDEX_UNSET) {
        clearPidList();
        currentMediaSourceIndex = nextWindowIndex;

        long positionMs = C.TIME_UNSET;
        long durationMs = player.getDuration();
        if (durationMs != C.TIME_UNSET) {
            positionMs = Math.min(positionMs, durationMs);
        }
        positionMs = Math.max(positionMs, 0);

        player.seekTo(nextWindowIndex, positionMs);

        
    } else {
        AltLog.d(TAG, "nextWindowIndex == C.INDEX_UNSET");
    }
}

[AquilesCanta]

I'm sorry but I can't really follow what's supposed to happen from the provided code but I don't have the resources to guide you to a full implementation either. Just a comment:

I don't think you should be using a ConcatenatingMediaSource at any point, but rather creating your own ProgressiveMediaSource fork that will handle the creation of new periods when a track change is detected.

What about on your side? Any updates on this?

No, and unfortunately, I don't expect any work to happen in this quarter.