Playback fails sometimes on android (top bit not zero and invalid atom length)

[jakubvojacek]

System info

Operating System: latest debian
Shaka Packager Version: latest master

Issue and steps to reproduce the problem

Packaging live unencrypted content crashes in android exoplayer randomly

Packager Command:

packager 
'in=udp://239.0.0.150:2010?reuse=1&buffer_size=10000000,stream=0,bandwidth=400000,init_segment=nsc-125_video_640_360.mp4,segment_template=video_640_360/nsc-125-$Number$.mp4' \
'in=udp://239.0.0.150:2010?reuse=1&buffer_size=10000000,stream=1,bandwidth=1000000,init_segment=nsc-125_video_720_410.mp4,segment_template=video_720_410/nsc-125-$Number$.mp4' \
'in=udp://239.0.0.150:2010?reuse=1&buffer_size=10000000,stream=2,bandwidth=4000000,init_segment=nsc-125_video_1920_1080.mp4,segment_template=video_1920_1080/nsc-125-$Number$.mp4' \
'in=udp://239.0.0.150:2010?reuse=1&buffer_size=5000000,stream=3,language=por, init_segment=nsc-125_audio_por-2874.mp4,segment_template=audio_por-2874/audio_por-2874-$Number$.mp4' \
'in=udp://239.0.0.150:2010?reuse=1&buffer_size=5000000,stream=4,language=eng, init_segment=nsc-125_audio_eng-2875.mp4,segment_template=audio_eng-2875/audio_eng-2875-$Number$.mp4' \
--time_shift_buffer_depth 15000 --hls_playlist_type LIVE \
--hls_master_playlist_output nsc-125.m3u8 \
--mpd_output nsc-125.mpd \
--fragment_duration 3.2 \
--segment_duration 3.2  \
--suggested_presentation_delay 5 \
--min_buffer_time 3.2 \
--minimum_update_period 5 \
--default_language=por \
--allow_approximate_segment_timeline \
--io_cache_size 3000000

Extra steps to reproduce the problem?

1. It happens randomly while playing the content on android exoplayer. It does not happen with shaka player. This issue was already reported on exoplayer by someone else but they're saying it is badly packaged content.
2. the url to the mpd is https://goo.gl/DMHAje
3. reported on exoplayer here Unexpected IllegalStateException: Top bit not zero When playing live dash stream google/ExoPlayer#2500 and here Top bit not zero when playing MP4 google/ExoPlayer#3384 and there are more cases like this, all of them marked as bad content by the exoplayer team

Errors we see in android

1. Unexpected exception loading stream java.lang.IllegalStateException: Top bit not zero: -6000
2. ExoPlayerImplInternal: Source error. com.google.android.exoplayer2.ParserException: Skipping atom with length > 2147483647 (unsupported).

Do you think this could be an issue with the shaka packager? I do not really understand what the errors mean but perhaps you could give us a hint whether to look for the issue in the packager or exoplayer.

Thank you

[kqyang]

@jakubvojacek It seems like exoplayer is complaining about the first bit in NAL unit type. We'll investigate if it is a packager bug or exoplayer bug or source content problem.

[kqyang]

It happens randomly while playing the content on android exoplayer. It does not happen with shaka player. This issue was already reported on exoplayer by someone else but they're saying it is badly packaged content.

@jakubvojacek Can you clarify what you mean by "randomly"? I am trying to reproduce the problem but wasn't able to. Do we need to play the content for hours to see it happening?

[jakubvojacek]

@kqyang It happened to me like 3x in five minutes of testing on exoplayer. I was not able to reproduce this with our ios player or dashsj / shaka

[kqyang]

@jakubvojacek Which version of ExoPlayer are you using?

[jakubvojacek]

the colleague responsible for android should wake up in a few hours and share more info but I searched the code and found this

    implementation 'com.google.android.exoplayer:exoplayer-core:2.8.2'
    implementation 'com.google.android.exoplayer:exoplayer-dash:2.8.2'
    implementation 'com.google.android.exoplayer:exoplayer-ui:2.8.2'

that looks like the exoplayer version.

[okycelt]

@jakubvojacek, that's correct, currently we're using 2.8.2

[kqyang]

We managed to re-produce the problem, after playing for more than 30 minutes. Will take a closer look at the problem!

[okycelt]

@kqyang, as @jakubvojacek said, it's quite random. Sometimes the playback fails within a few minutes and keeps failing, sometimes it takes longer.

Thank you.

[kqyang]

There is indeed a problem in trun box. Here is a list of entries I extracted from a trun box:

Entry{duration=3000,...}
Entry{duration=3000,...}
Entry{duration=9000,...}
Entry{duration=-6000,...}
Entry{duration=3000,...}
Entry{duration=6000,...}
Entry{duration=3000,...}

Seems like the problem originates from the source stream, where samples are not put in the correct order.

@okycelt @jakubvojacek Can you record a few hours of contents and send to shaka-packager-issues@google.com? It will help us find out where the problem is and potentially workaround the problem. Since the problem exists on all resolutions, you can record the lowest resolution.

[jakubvojacek]

@kqyang just sent the email, thank you for your kind assistance 👍

[kqyang]

@jakubvojacek So there is an indeed a problem in the source stream. In the file you sent to me, packet 13740594 has a pts of 6126057840 (pts1) while the next sample, packet 13740622, has a pts of 6126048840 (pts2), which is < pts1.

Neither samples have dts.

Per 13818-1 specification, A decoding_timestamp (DTS) shall appear in a PES packet header if the decoding time differs from the presentation time.

There are no dts in the above two samples, so packager infers that dts = pts. The duration is calculated from the difference in dts, which results in the negative duration for sample 1 as the next sample has a smaller dts.

I wonder how is the source stream produced. It may be a valid stream, just missing dts values.

We could workaround the problem by dropping the samples with negative timestamp, but that may result in other problems though during playback.

[jakubvojacek]

@kqyang We will ask the operator to check with their multicast provider. Meanwhile, what other problems are you referring to? Will it result in gaps in the content? Would it be possible to add a warning in the packager output each time a sample like this occurs and drop it?

Thank you

[kqyang]

what other problems are you referring to?

There may be decoding problems if there are other frames depending on the dropped frame. I don't see any playback problems though on the packaged content from your stream in either ExoPlayer or Shaka Player.

Will it result in gaps in the content?

There are actually a small overlap instead of a gap because dropping the frame with negative duration results in increase in total duration.

Entry{duration=3000,...}
Entry{duration=3000,...}
Entry{duration=9000,...}
Entry{duration=-6000,...}
Entry{duration=3000,...}
Entry{duration=6000,...}
Entry{duration=3000,...}

Would it be possible to add a warning in the packager output each time a sample like this occurs and drop it?

Yes, we will.

[kqyang]

@jakubvojacek The workaround has been submitted. We decided not to drop the frame but modifying the duration instead to avoid possible decoding problem. Please try and let us know if it works for you.

[jakubvojacek]

@kqyang we have not experienced the issue since the update, seems like it's working perfectly. Thank you 👍

[kqyang]

Cherry-picked to v2.2.1.