Multiple Event/Emsg-related problems

[nicoweilelemental]

Environment

• The MPD passes the DASH-IF Conformance Tool on https://conformance.dashif.org/
• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream
• The issue observed is not mentioned on https://github.com/Dash-Industry-Forum/dash.js/wiki/FAQ
• The issue occurs in the latest reference client on http://reference.dashif.org/dash.js/ and not just on my page

• Link to playable MPD file (live stream, 2s segments): https://c4af3793bf76b33c.mediapackage.us-west-2.amazonaws.com/out/v1/769dc73e096b47b290bfc3f7278f28f9/index.mpd
• Dash.js version: 3.1.3
• Browser name/version: Chrome 85
• OS name/version: Win10

Steps to reproduce

1. Please provide clear steps to reproduce your problem
   Load stream in dash.js or load Crafter player for more debugging capability (https://videosuite.craftercms.org)

2. If the bug is intermittent, give a rough frequency if possible
   Problems are permanent

Observed behaviour

We are injecting KLV metadata in emsg boxes v1 (one emsg box per KLV message). This is done as per the MISB ST 1910 spec. The data rate can be as high as one KLV message per video frame.

1. Emsg don't trigger stream event in VOD
   When a VOD extract of the live stream is played, no event is triggered like with the live stream.
   https://c4af3793bf76b33c.mediapackage.us-west-2.amazonaws.com/out/v1/769dc73e096b47b290bfc3f7278f28f9/index.mpd?start=2020-10-12T14:30:00-07-00&end=2020-10-12T14:35:00-07-00
   We suspect that there might be a problem with events management in VOD streams.

2. Multiple emsg boxes content are aggregated inside a single event payload
   This is visible in the Crafter player. The left counter ("emsg messages") is the number of events received and the right counter is the number of KLV samples found inside the events. The two counters should have the same value if each emsg box was treated as a single event.
   This can be further shown by typing "printKlvEvents=1" in the javascript console of the Crafter player. It will show the detail of the received messages. The number of lines in the messageData section of each Event wil show how many emsg boxes are contained in a single event: one emsg box is 2 lines of data, and the Events including 3 emsg boxes have 6 lines of data.
   We suspect that there might be an aggregation of multiple emsg boxes contents into a single event messageData.

3. Segment duration influences extracted events rate
   Same stream but with 30s segments shows a much higher number of events being extracted
   https://c4af3793bf76b33c.mediapackage.us-west-2.amazonaws.com/out/v1/e0c840c123cb48429edca086064d299f/index.mpd
   We suspect that this is correlated to the previous point, meaning that the events signaling cycle seems to not be based on the number of emsg boxes present in the segments, but on a timing logic that is specific to the player.

4. Emsg boxes v1 don't seem to be supported
   We use emsg boxes v1 (and not v0). But when we look at the details of the Events (using the "printKlvEvents=1" console command in the Crafter player), we see that the Events have got a presentationTime: NaN field value. We suspect that this is either the result of the concatenation of multiple emsg boxes content into a single Event, or caused by a lack of support for emsg boxes v1, where the data fields are not ordered the same way in the payload.

A diagnosis would be highly appreciated. Thanks!

[dsilhavy]

@nicoweilelemental

1. Can you please reupload the VoD example, the url results in 404. Maybe there is a timing issue related to presentationTimeOffset or sth similar, got to check this.
2. This has to be investigated further. dash.js is relying on an external box parsing library in order to extract emsg data. The message_data field, as parsed by this external library, already shows different length for the message_data field. I checked with Isoviewer and the parsing result is the same. The length of the messageData field differs. I expect the behavior of dash.js to be correct, however we should confirm this.
3. dash.js checks every 100ms if events need to be triggered. There is a 300ms threshold, which means events are still triggered if:
   if (Math.abs(calculatedPresentationTimeInSeconds - currentTime) < REMAINING_EVENTS_THRESHOLD)
4. Emsg v1 is supported timing wise, we "calculatedPresentationTime" in order to handle version 0 and 1 with the same logic. Can you check if calculatedPresentationTime contains the value you would expect?

[dsilhavy]

Update on 1. Seems to be related to wrong/no usage of the presentationTimeOffset when calculating the presentation time. This should be fixed in dash.js.

[nicoweilelemental]

Hi Daniel, thanks for looking into this.

I compared our stream with the stream from MISB https://www.mistandards.org/demo/cmaf/data/H264/cmafsample.mpd and it appears that while each emsg box from the MISB stream has got a distinct presentation_time value, in our stream I can find examples where multiple emsg boxes have got the same presentation_time value, which can explain problem 2. I think we need to fix it and see if it changes the behaviors on problems 3. and 4.

The MISB stream can be interesting as a comparison point on problem 1. too. They have a dash.js-based player (https://www.mistandards.org/demo/cmaf/cmaf-viewer/) where the stream plays fine the $Number$-based stream. I guess it confirms your diagnosis about presentationTimeOffset and $Time$-based streams being problematic.

[nicoweilelemental]

Actually it looks like the fact that emsg boxes reuse the same id value across segments is the main problem for live streaming, now that we have verified all the other points in our implementation. When receiving a new emsg box, the player treats it as an update, as the emsg id already exists in the timeline/memory. Hence this question: do the emsg box id (or the equivalent event stream id after parsing of the emsg boxes) stay forever in the player memory, or are the events flushed out when the timeshift buffer depth is reaching its limit?

[dsilhavy]

The events are flushed out either after being expired or after being triggered. This includes:

• event.duration + event.presentationTime < currentVideoTime : for instance when the user seeks over the event
• event.presentationTime > period.start + period.duration : The presentation time of the event is out of its period boundary
• The event has expired: currentVideoTime - presentationTimeThreshold > calculatedPresentationTimeInSeconds: In my opinion the current logic for this is wrong, it should take the duration of the event into account
• If the event was triggered(onStart mode) it is removed

Regarding updating/replacing existing events: This is only the case for inline events. Inband events which are already exist (same id) are ignored:

Inline : Replacing existing events

          if (values) {
                for (let i = 0; i < events.length; i++) {
                    let event = values[i];
                    inlineEvents[event.id] = event;
                }
            }

Inband: : Ignoring event updates

 if (!(event.id in inbandEvents)) {

}

I sent a longer mail to the Event TF which includes exactly this question: Are we supposed to replace existing inband events which have not been started? Or do we ignore them? In my opinion the behavior for inline and inband events should be consistent. I haven't received any responses on my questions yet, will try to bring this up in the next Event TF call.

[nicoweilelemental]

@dsilhavy Thanks for the insight, this is as complex as I was expecting. Apparently we have hit another player rule, which is the single execution per event id (unless this is the inband event updates that are ignored), but that's clearly because our ids repeat over time, and this is not spec compliant. See Gary's (MISB) diagnosis below:
OK, I believe I found the (real) problem. The emsg id's in each segment restart at zero, and dash.js is recognizing duplicate id's and discarding them. Periodically the buffer clears and a new bunch are accepted, but then the next following few segments' emsg packets are again discarded.
To confirm this, I modified dash.js to replace their emsg id's with a sequence of unique ids across segments, and I can see their KLV updating continuously.
In the DASH spec, the id is allowed to be repeated but only for actual duplication, and the extras can be discarded, so I think the player is handling it properly:
(DASH par. 5.10.3.3.4)
...
id: a field identifying this instance of the message. Messages with equivalent semantics shall have the same value, i.e. processing of any one event message box with the same id is sufficient.

As the ids need to be unique as long as they are in the playback window, there is no reason why the inline and inband updates should behave differently. Allowing updates in both cases (if the event.presentationTime hasn't been reached yet) seems to be the safest approach as you can always meet cases where the first message is erroneous and an update is sent to fix it.

As regards problem 1. (VOD and no usage of PTO), the best I can do in terms of VOD sample stream is to propose this live URL with time parameters that actually generates a VOD manifest: https://c4af3793bf76b33c.mediapackage.us-west-2.amazonaws.com/out/v1/769dc73e096b47b290bfc3f7278f28f9/index.mpd?start=2020-12-02T15:30:00-07-00&end=2020-12-02T15:35:00-07-00
It should work for the next 14 days, and after this timeframe we'd need to modify the start/end parameters values with a more recent date. I hope to come up with static VOD vectors at some point but probably not before end of January.

[dsilhavy]

@nicoweilelemental I did some changes to the event timing model in dash.js in #3471 . They should be reflected in the nightly branch: http://reference.dashif.org/dash.js/nightly/samples/dash-if-reference-player/index.html

I can see events being dispatched using your Live to VoD sample from above. Can you confirm that the timing is correct?

Sidenote: I feel like a high amount of events can crash the player. We might need to find a less resource consuming way to manage and dispatch them.

[nicoweilelemental]

@dsilhavy Thanks for the fix, the flow of events looks good. I'll need to get the nightly library integrated into a partner player to verify the KLV display, as there is a specific processing library to use for that.

Regarding the amount of events: did you manage to trigger a crash or is it just an assumption?

[dsilhavy]

I saw playback stalling when printing events to the developer console. However, it worked fine when closing the developer console. Looking forward to the findings of your partner

[nicoweilelemental]

@dsilhavy Quick clarification question: if I send two different emsg boxes with the same 'id' field value but not the same 'value' field value, it will result in two different events, and not in an overwrite operations of a single event, right?

[dsilhavy]

@nicoweilelemental No, the event will only be handled once if the id field value is the same. However, it seems you are right this should be different. From 23009-1:

Messages with the same id within the scope of the same scheme_id_uri and value pair are equivalent , i.e. processing of any one event message box with the same id is sufficient

We need to consider the scope here (scheme_id + value). We will fix this, thanks for pointing it out.

[dsilhavy]

@nicoweilelemental The problem with events using the same id fields and overwriting each other despite different value fields should be fixed in #3504. It was merged into the nightly build

[nicoweilelemental]

Thanks @dsilhavy , much appreciated!

[dsilhavy]

@nicoweilelemental Do you think we can close this issue? Or did your tests show additional issues?

[nicoweilelemental]

HI @dsilhavy I think we can close it. I need to run additional tests once we will have modified our implementation for KLV. Will open a new ticket if I find further problems. Thanks

[kjerbi]

Hi @nicoweilelemental
Thanks for the details above.
As the link of the mpd returns a 403 can you share any updated link to an mpd or even a sample of a cmaf chunk with emsgs containing klv?
Thank you in advance