Add support for HLS AUDIO group

[guoen21]

Use case description

Currently the audio group is unavailable, when the player adaptive switch to one alternative video track, the audio does not switch the audio track defined on "AUDIO" of this video track.

Here are one example, the manifest defined 6 video tracks with main and backup stream, for each video track defined the audio group. Also defined the 18 audio tracks, for each audio group it has away, default and french options.

Currently the playback is on main/video/1512000/index.m3u8 with main/audio/192000_away/index.m3u8 (audio groupId - main-audio-192000)
if the playback switch to main/video/600000/index.m3u8 because of the bandwidth reason, and the player still play the main/audio/192000_away/index.m3u8.

But normally the audio track should be main/audio/128000_away/index.m3u8 (audio groupId - main-audio-128000)
I think it may be a place that can be improved.

#EXTM3U
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA:TYPE=AUDIO,URI="main/audio/64000_away/index.m3u8",GROUP-ID="main-audio-64000",NAME="away",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="main/audio/128000_away/index.m3u8",GROUP-ID="main-audio-128000",NAME="away",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="main/audio/192000_away/index.m3u8",GROUP-ID="main-audio-192000",NAME="away",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="main/audio/64000/index.m3u8",GROUP-ID="main-audio-64000",NAME="default",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="main/audio/128000/index.m3u8",GROUP-ID="main-audio-128000",NAME="default",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="main/audio/192000/index.m3u8",GROUP-ID="main-audio-192000",NAME="default",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="main/audio/64000_fr/index.m3u8",GROUP-ID="main-audio-64000",NAME="French",LANGUAGE="fra",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="main/audio/128000_fr/index.m3u8",GROUP-ID="main-audio-128000",NAME="French",LANGUAGE="fra",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="main/audio/192000_fr/index.m3u8",GROUP-ID="main-audio-192000",NAME="French",LANGUAGE="fra",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"

#EXT-X-MEDIA:TYPE=AUDIO,URI="backup/audio/64000_away/index.m3u8",GROUP-ID="backup-audio-64000",NAME="away",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="backup/audio/128000_away/index.m3u8",GROUP-ID="backup-audio-128000",NAME="away",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="backup/audio/192000_away/index.m3u8",GROUP-ID="backup-audio-192000",NAME="away",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="backup/audio/64000/index.m3u8",GROUP-ID="backup-audio-64000",NAME="default",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="backup/audio/128000/index.m3u8",GROUP-ID="backup-audio-128000",NAME="default",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="backup/audio/192000/index.m3u8",GROUP-ID="backup-audio-192000",NAME="default",LANGUAGE="eng",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="backup/audio/64000_fr/index.m3u8",GROUP-ID="backup-audio-64000",NAME="French",LANGUAGE="fra",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="backup/audio/128000_fr/index.m3u8",GROUP-ID="backup-audio-128000",NAME="French",LANGUAGE="fra",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,URI="backup/audio/192000_fr/index.m3u8",GROUP-ID="backup-audio-192000",NAME="French",LANGUAGE="fra",AUTOSELECT=YES,DEFAULT=NO,CHANNELS="2"

#EXT-X-STREAM-INF:BANDWIDTH=1965637,AVERAGE-BANDWIDTH=1694815,FRAME-RATE=25,PROGRAM-ID=1,CODECS="avc1.640029,mp4a.40.2",RESOLUTION=1280x720,AUDIO="main-audio-192000"
main/video/1512000/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1965637,AVERAGE-BANDWIDTH=1694815,FRAME-RATE=25,PROGRAM-ID=1,CODECS="avc1.640029,mp4a.40.2",RESOLUTION=1280x720,AUDIO="backup-audio-192000"
backup/video/1512000/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=936650,AVERAGE-BANDWIDTH=767625,FRAME-RATE=25,PROGRAM-ID=1,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=640x360,AUDIO="main-audio-128000"
main/video/600000/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=936650,AVERAGE-BANDWIDTH=767625,FRAME-RATE=25,PROGRAM-ID=1,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=640x360,AUDIO="backup-audio-128000"
backup/video/600000/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=552324,AVERAGE-BANDWIDTH=446593,FRAME-RATE=25,PROGRAM-ID=1,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=512x288,AUDIO="main-audio-64000"
main/video/348000/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=552324,AVERAGE-BANDWIDTH=446593,FRAME-RATE=25,PROGRAM-ID=1,CODECS="avc1.42c01f,mp4a.40.2",RESOLUTION=512x288,AUDIO="backup-audio-64000"
backup/video/348000/index.m3u8

[andrewlewis]

@christosts Please could you take a look? Thanks.

[guoen21]

Can anyone answer this question? Thanks.

[christosts]

Yes, this is a part of the HLS spec that ExoPlayer does not support at the moment, and we have not prioritized implementing. I am keeping this issue in the backlog, but I can't make any commitments on when we'll schedule working on this. [Internal ref: b/128767128]

In the mean time, do you need assistance how to implement this with custom components?

[guoen21]

Thanks very much for your reply, I will be very grateful if I can get your assistance.

I also try to implement the audio group feature based on release 2.14.2, it seems that a lot of code has been changed. These are some key points:

1. Append group info to format when parsing master m3u8 and preparing tracks format for each period.
2. Collect the track blacklist info globally. For video track, it may be blacklisted because of the audio group constraint.
3. Apply group constraint when reselect tracks in DefaultTrackSelector, also enhance adaptive selection:

• Support to set initial selected index
• Support to determine the best selected index with current bandwidth and global blacklist information.

4. When adaptive video track is changed, we may need reselect tracks to change the audio track synchronously.

I can make one commit and PR to review on Exoplayer for audio group feature. But not sure which branch is most suitable.

[christosts]

Hi and apologies for the delay. In case it helps, here is some guidance on how to write a custom track selection that adjusts its audio track selection based on group(s) of the selected video track, using the existing APIs available on the player. The following is for version 2.16. I tried to keep the text below thorough, yet short :)

Group info is already appended in the Format associated with a track. In Format.metadata.entries, there is a Metadata.Entry of type HlsTrackMetadataEntry. For audio tracks, their group ID (if any) is stored in HlsTrackMetadataEntry.groupId. For video tracks, iterate
HlsTrackMetadataEntry.variantInfos and inspect audioGroupId to see the group IDs associated with the video track.

With the existing APIs, you can do the following:

• Create a player using a DefaultTrackSelector that uses a custom ExoTrackSelection.Factory.
• In your ExoTrackSelection.Factory.createTrackSelections(), you can find the audio group IDs each video track is associated with, and the group ID of each audio track. From the Definitions [] argument, find the Definition for video and audio and then:
  • For each video track, iterate HlsTrackMetadataEntry.variantInfos and inspect VariantInfo.audioGroupId.
  • For each audio track, inspect HlsTrackMetadataEntry.groupId.
• At this point you should be able to have a mapping of [video track index: {audio group IDs}], and [audio track index: group ID].
• ExoTrackSelection.Factory.createTrackSelections() returns an array of ExoTrackSelection, containing an ExoTrackSelection per track type (one for audio, one for video, one for subtitles, etc). For video and audio, you need to create custom ExoTrackSelection instances such that when an audio track is requested, the audio track selection will consult the video track selection for the equivalent media segment.
  • There is a strong assumption here that your playlist has independent segments (it should!) and you can derive which video segment matches to a certain audio segment (e.g. their numbers match?).
• In order to avoid rewriting an adaptive bitrate selector, you might want to forward the video track selection itself to the in-library AdaptiveTrackSelection implementation. Therefore, one option is to have your ExoTrackSelection.Factory.createTrackSelections() forward the call to AdaptiveTrackSelection.Factory.createTrackSelections() and then wrap/replace the returned audio and video track selection instances:
  • Wrap/replace the video ExoTrackSelection so that you keep track what track index was selected per media segment (see updateSelectedTrack() and getSelectedIndex() in ExoTrackSelection)
  • Wrap/replace the audio ExoTrackSelection so that when a track index is requested for a segment (updateSelectedTrack()), it consults video track selection to figure out the video track index was selected for the equivalent segment, and therefore limit the audio tracks to the associated groups.

Thanks

[guoen21]

Thanks very much for your reply.