Fix playback of I-FRAME only variants in Apple's advanced FMP4 stream

[AquilesCanta]

To reproduce, simply play the "Apple master playlist advanced (FMP4)" sample from the demo app and select (using the track selection UI) an I-FRAME only track.

Note: I-FRAME only variants are only accessible through track selection overrides.

[stevemayhew]

@AquilesCanta what is happening is the FragmentedMp4Extractor is returning exiting the read() loop (returning RESULT_END_OF_INPUT), but ExtractorInput.getPosition() returns the length of the stream (like it is not fully read).

This makes the subrange computed on HlsMediaChunk#L365 invalid (as the left over value in nextLoadPosition is incorrect.

I have sample mp4 files that seem to re-create the issue, if you are ok with it I will add this to the FragmentedMp4Extractor test cases and submit a pull request if it fixes the issue.

Thanks for catching it!.

[AquilesCanta]

I'm still not clear how the nextLoadPosition is involved. I thought nextLoadPosition was relevant when there was an error while reading the media. The error I ran into when I tried the media was an end of input exception. But when I added a catch to ignore that exception, and treat the chunk as completed (In which case nextLoadPosition should not matter), playback still failed. Any PRs are welcome though. If just for reference you can explain the phenomenon, it would be really useful. To be honest I didn't really dig that much. It's possible it's really simple.

[stevemayhew]

Well, nothing is ever really simple ;-). I'm not clear on the history (looks very long back) on the comments on line 355 but it seems the range in the dataSpec is (was) being used to partially fetch segments this would seem to overlap/conflict with the fact that all the iframe only segments are already byte-ranged to a single sample.

At the completion of parsing of one of these ranged segments input.getPosition() - dataSpec.absoluteStreamPosition is non-zero so that messes up the next byte-range fetch.

I will figure it out, right now we are updating our release to merge in your release tag r2.11.6 (which looks like the branch missed the cutoff to include our pull i-frame only support anyway).

[mdobrzyn71]

I'm sharing here what I've found so far on this issue. I'm not trying to propose any workaround/fix just yet.

Apple's Fmp4 IFrame-only stream is simply a sequence of moof->mdat->moof->mdat->... atoms, where each pair (moof->mdat) contains entire GOP (similar to many regular video stream). Frames other than reference ones are also in there, however, the Apple's stream generator ranges them out in the HLS playlist. During playback, once initialized with 'moov' atom, the Fmp4Extractor, is fed with data chunks that always start with the 'moof' box and the following 'mdat' box, truncated past the first sample, which is the GOP opening IFrame. Upon 'moof' container reception the extractor is parsing the entire GOP as it doesn't know at this point that only the first IFrame is of player's interest. Besides minor CPU overhead, that doesn't seem to be an issue as the parser output lands in the temporary internal data structures such as TrackBundle. Once all the metadata is parsed and the 'mdat' box header has arrived, the extractor tries to extract out the first sample, which should be the IFrame. That part also works fine for Apple's IFrame stream as the byte range includes the 'mdat' part containing desired frame. The Fmp4Extractor.read(...) method is written to read one sample at a time so, after getting the IFrame data it saves its state and returns. Stream position is saved too. The value returned indicates whether the extractor expects to be called again to continue extracting from that spot. In Apple's IFrame stream case it would probably be best to say 'no-more' at this point but the current code doesn't check if the actual stream position is just few bytes away from the end of the byte range and it tries to keep going. Method HlsMediaChunk.feedDataToExtractor(...) calls Fmp4Extractor.read(...) again and that call gets an IOexception as it tries to read next sample beyond the actual byte range. Unfortunately, this exception is handled by the Loader.run() method, who applies its internal re-try logic in such a case without letting anyone else know about it. Ultimately 'loadable' object i.e. HlsMediaChunk gets loaded again (loadable.load()->loadMedia()->feedDataToExtractor()) and this time we hit assetrs for incorrect byte range update and other follow-up errors which fail the playback.

The investigation is in progress, I'll soon add a unit test that will try to reproduce this issue.

[stevemayhew]

@AquilesCanta Basically the issue is Apple is referencing parts of the base MP4 file in both the playlist and the i-frame playlist.

The playlist segments contain an entire MDAT (the full GOP), the i-Frame only playlist truncates the MDAT after the first sample (the IDR).

Basically, Apple's Fmp4 IFrame-only stream, just like their TS streams, are produced by byte-ranging out part of the main stream.

For example:

1. Variant playlist — /v7/prog_index.m3u8

#EXT-X-MAP:URI="main.mp4",BYTERANGE="720@0"
#EXTINF:6.00000,	
#EXT-X-BYTERANGE:3381026@720
main.mp4
#EXTINF:6.00000

1. Matching iFrame — /v7/iframe_index.m3u8

#EXTINF:2.00000,	
#EXT-X-BYTERANGE:44379@720
main.mp4

The BYTERANGE in the first i-Frame starts at the offset of the first MOOV in main.mp4 but includes the entire i-Frame sample but ends before the end of the
following MDAT (ends at 1128636).

The Range includes the MOOV -> MOOF, but not all of the following MDAT

You can see this by downloading main.mp4 and using the handy Online MP4 Parser to parse it out. It
shows this for the first MDAT following the opening MOOF

FIELDS	VALUES
Box Type	mdat
Box Size	1126356
Start position	2280
End position	1128636

I'll let Maciej (@mdobrzyn71 ) comment further on how this is causing the Fmp4Extractor heartaches.

[AquilesCanta]

Hi @mdobrzyn71 and @stevemayhew, I think I got that far. However, I remember trying to catch and ignore EOFExceptions coming from Fmp4Extractor.read(), by treating them as normal end of input. However, the resulting behavior didn't change, so I assumed there was something else involved.

Please feel free to try it yourselves: In HlsMediaChunk.feedDataToExtractor you can try catching any exceptions thrown and treating them as "end of extraction"l, and see if the file plays normally. If it doesn't, we need to understand why. That will get us closer to the root cause.

[mdobrzyn71]

Hi @AquilesCanta , @stevemayhew

I've implemented a temporary patch that allows Apple's IFrame-only stream to be played : p-fix-apple-iframe-bug

Although quick & working, this patch has some severe performance cons:

1. it destroys the FragmentedMp4Extractor object every time new I-Frame segment is being loaded.
2. media initialization section is loaded and extracted when the above is happening.

An ideal implementation should NOT destroy the FragmentedMp4Extractor object. Instead, it should zero out all track fragments collected so far, retaining the track definitions. In other words, it should go back to the state right after extracting MOOV atom.

As I mentioned, method FragmentedMp4Extractor.seek() does almost all what's needed. Unfortunately, it can't be called from the HlsMediaChunk object.

Two options:

• Add abstract method, say resetTracks() in the Extractor class and implemented in the FragmentedMp4Extractor
• Handle the error internally in the FragmentedMp4Extractor and return early if it runs out of input (hard)

question to @ojw28 : Oliver, what about initialization of DRM for partial segments like i-Frame?

[AquilesCanta]

The fix has been merged. Please reach back to us if you find any further issues with I-FRAME-ONLY variants playback.