Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for AVC fmp4 fragments found in Apple Advanced HEVC/H.264 stream #3274

Merged
merged 2 commits into from Dec 7, 2020

Conversation

robwalch
Copy link
Collaborator

@robwalch robwalch commented Dec 3, 2020

This PR will...

Add support for AVC fmp4 fragments found in Apple Advanced HEVC/H.264 stream

Why is this Pull Request needed?

Playback of https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8 without JavaScript exceptions.

When playing this stream, mp4-tool functions such as getStartDTS, offsetStartDTS , and getDuration would throw exceptions when encountering track fragment boxes for unknown track types* not accounted for in the init data (initData[id] is undefined).

These changes ignore those track fragments so that we only deal with the audio and video fragments we're parsing time/duration for.

*Track types added to init data:

const type: HdlrType = { soun: ElementaryStreamTypes.AUDIO, vide: ElementaryStreamTypes.VIDEO }[hdlrType];
if (type) {

Are there any points in the code the reviewer needs to double check?

Replaced some array methods where appropriate (map > reduce or forEach).

This is also broken in v0.x, so a patch in master should be made for v0.15.

Demo:

https://deploy-preview-3274--hls-js-dev.netlify.app/demo/?src=https%3A%2F%2Fdevstreaming-cdn.apple.com%2Fvideos%2Fstreaming%2Fexamples%2Fbipbop_adv_example_hevc%2Fmaster.m3u8&demoConfig=eyJlbmFibGVTdHJlYW1pbmciOnRydWUsImF1dG9SZWNvdmVyRXJyb3IiOnRydWUsInN0b3BPblN0YWxsIjpmYWxzZSwiZHVtcGZNUDQiOmZhbHNlLCJsZXZlbENhcHBpbmciOi0xLCJsaW1pdE1ldHJpY3MiOi0xfQ==

Checklist

  • changes have been done against master branch, and PR does not conflict
  • new unit / functional tests have been added (whenever applicable)
  • API or design changes are documented in API.md

@robwalch
Copy link
Collaborator Author

robwalch commented Dec 3, 2020

Hi @kanongil,

You were the last person to provide enhancements to fmp4 parsing. Would you mind reviewing these changes? They aren't final as the parsed fragment durations are incorrect. Any pointers would be much appreciated! Thank you.

@@ -354,10 +366,17 @@ export function getDuration (data: Uint8Array, initData: InitData) {
// get the track id from the tfhd
Copy link
Collaborator

@itsjamie itsjamie Dec 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

The file here advertised with the major brand mp42, not the CMAF brand, and is multi-trun. So we have durations split across multiple track runs on the same track. I see below you've started to account for this, but it'll be more parsing issues like this. It may make sense in the future to pull apart the parser to make a brand-specific parser for CMAF since it is intended to be much simpler.

src/utils/mp4-tools.ts Outdated Show resolved Hide resolved
return baseTime / scale;
}
// eslint-disable-next-line no-console
console.warn(`getStartDTS: No fmp4 track found for tfhd id ${id}`);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warning is occurring for track-id 2 on the test content because we do not handle the clcp style of hdlr. It's a closed caption track.

if (hdlr) {
const hdlrType = MP4Demuxer.bin2str(hdlr.data.subarray(hdlr.start + 8, hdlr.start + 12));
let type = { 'soun': 'audio', 'vide': 'video' }[hdlrType];
if (type) {
// extract codec info. TODO : parse codec details to be able to build MIME type
let codecBox = MP4Demuxer.findBox(trak, ['mdia', 'minf', 'stbl', 'stsd']);
if (codecBox.length) {
codecBox = codecBox[0];
let codecType = MP4Demuxer.bin2str(codecBox.data.subarray(codecBox.start + 12, codecBox.start + 16));
logger.log(`MP4Demuxer:${type}:${codecType} found`);
}
result[trackId] = { timescale: timescale, type: type };
result[type] = { timescale: timescale, id: trackId };
}
}

MP4Box.js output of track info for test content

tkhd

hdlr

Copy link
Collaborator Author

@robwalch robwalch Dec 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got rid of that. Now fragment boxes for those tracks are ignored. Thanks so much for pointing this out!

@robwalch robwalch added this to the 1.0.0 milestone Dec 4, 2020
@robwalch robwalch added this to Top priorities in Release Planning and Backlog via automation Dec 4, 2020
@robwalch robwalch moved this from Top priorities to v1.0.0 in Release Planning and Backlog Dec 4, 2020
@robwalch robwalch merged commit d1fc2c5 into feature/v1.0.0 Dec 7, 2020
@robwalch robwalch deleted the bugfix/fmp4-avc-640020 branch December 7, 2020 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants