DASH: Improve handling of EMSG tracks

[chtr-dev-msummsers]

Hello,

Our project has a requirement to report some 3rd party ID3 metadata contained within our DASH streams.
This metadata is in the format of <InbandEventStream> XML tags inside audio <AdaptationSet> of our manifests in addition to emsg atoms in the fMp4.

Example from manifest:
<AdaptationSet lang="en" segmentAlignment="true" mimeType="audio/mp4" startWithSAP="1">
<InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1" />
</AdaptationSet>

For the majority of the streams, everything works as expected and we are able to see EMSG output from EventLogger (eg MetadataOutput.onMetadata() is called from MetadataRenderer ):
com.google.android.exoplayer2.demo D/EventLogger: EMSG: scheme=www.3rd-party.com:id3:v1, id=0, durationMs=728, value=1

However for some streams, MetadataOutpout.onMetadata() is not fired and no EMSG output is observed in the logs. We are unsure whether it is a problem with our streams, or a problem with some ExoPlayer logic. Hence, we are opening this issue as a question rather than a bug or feature request until root cause has been identified.

We are leaning toward some ExoPlayer issue, as our web devs have reported that the Shaka player is able to output the events from the same streams as expected. Additionally, as stated in this issue's title, ExoPlayer's FragmentedMp4Extractor successfully encodes the EventMessage with appropriate messageData when processing the mp4:
EventMessage eventMessage = new EventMessage(schemeIdUri, value, durationMs, id, messageData);
ParsableByteArray encodedEventMessage = new ParsableByteArray(eventMessageEncoder.encode(eventMessage));

From debugging the ExoPlayer source code, the main difference in behavior we see is that inside MetadataRenderer.read() function, the call to readSource() never returns C.RESULT_BUFFER_READ for the streams that do not fire this event, and so no pendingMetadata is set. But since there are no warnings or errors, it is difficult to understand where and why things are going wrong. We are currently on ExoPlayer version 2.14.2, and have tried the latest version 2.15.1 as well as the previous version we were on, 2.12.3. We can reproduce this behavior using the same streams within the ExoPlayer demo app.

I have sent sample streams to dev.exoplayer@gmail.com that demonstrate expected vs problematic behavior.
Any insight would be greatly appreciated.
Thank you for your time.

[ojw28]

Thanks for the detailed report and easy reproduction steps! I think the media is valid. I can explain what the difference is between the working and broken samples provided.

If you look at the manifests, you can see there's a difference in structure between the two:

Working MPD
-----
Video
    [No in-band event stream]
Audio
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>
Text
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>

Broken MPD
-----
Video
    <InbandEventStream schemeIdUri="urn:mpeg:dash:event:2012" value="1"/>
Audio
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>
Text
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>

The problem is that ExoPlayer will only output EMSG data from one of the adaptation sets, and it picks the first one that declares at least one InbandEventStream. So in the "broken" case, the metadata that gets output is the schemeIdUri="urn:mpeg:dash:event:2012" metadata, not the www.3rd-party.com:id3:v1 metadata that you're looking for. Furthermore, it's pretty hard for you to specify a preference to select to receive the www.3rd-party.com:id3:v1 metadata instead, as things are in ExoPlayer today.

We should look at improving how ExoPlayer does things here, although it's likely going to be faster for you to work around the problem by tweaking your content, if feasible. I think that all of the following would work (Note: I'm not really saying these are great solutions; I'm just outlining what you could do to get things working):

1. Add the 3rd party EMSG data to the video streams as well. You'll receive all of the
   EMSG data in the video adaptation set (i.e., both `urn:mpeg:dash:event:2012` and
   `www.3rd-party.com:id3:v1` metadata). It should be easy for you to filter out just the
   `www.3rd-party.com:id3:v1` metadata from there.

Video
    <InbandEventStream schemeIdUri="urn:mpeg:dash:event:2012" value="1"/>
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>
Audio
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>
Text
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>

2. Move the `urn:mpeg:dash:event:2012` metadata to the audio stream:

Video
    [No in-band event stream]
Audio
    <InbandEventStream schemeIdUri="urn:mpeg:dash:event:2012" value="1"/>
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>
Text
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>

3. Swap the order of the adaptation sets in the manifest so audio comes first:

Audio
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>
Video
    <InbandEventStream schemeIdUri="urn:mpeg:dash:event:2012" value="1"/>
Text
    <InbandEventStream schemeIdUri="www.3rd-party.com:id3:v1" value="1"/>

[ojw28]

Things to look at on the ExoPlayer side:

• Make it easier for apps to select which metadata track is output. For example, by getting the schemeIdUris into the track somehow, and allowing selection via DefaultTrackSelector.
• Whether it makes sense to expose urn:mpeg:dash:event:2012 as a track at all, given these events are handled internally within the player. Note that if we don't expose it, we do still need to make sure that the internal handling continues to work correctly

[chtr-dev-msummsers]

Thank you for the quick response!
I am glad we could make it easy for you to identify the problem, and am looking forward to the fix.
You weren't kidding ExoPlayer doesn't make it easy to select a preferred adaptation set for EMSG.
Took a stab at trying the suggested work-arounds by setting a custom manifest parser on the media source factory, and the easiest option appears to be resorting the adaptation sets, in ascending order by type (C.TRACK_TYPE_AUDIO has lower value than C.TRACK_TYPE_VIDEO)