Refactor transmuxing data flow #199

johnBartos · 2019-02-25T20:42:07Z

This PR will...

Remove all events from the transmuxing pipeline, aside from ERROR and FRAG_DECRYPTED
Typescripts the following:
- tsdemuxer
- mp4-remuxer
- mp4demuxer
- passthrough-remuxer
- demuxer (renamed to transmuxer-interface)
- demuxer-inline (renamed to transmuxer)
- demuxer-worker (renamed to transmuxer-worker)
Refactor the demuxer's append method to a new method: demux:
- Accepts binary data
- Synchronously returns the demuxed data as a DemuxerResult object when unencrypted
- Asynchronously returns a DemuxerResult promise when data is SAMPLE-AES encrypted
Refactor the remuxer's remux method:
- Accepts values from a DemuxerResult object
- Synchronously returns a RemuxerResult object
Refactor demuxer-inline to transmuxer
- Owns both a demuxer and remuxer
  - The demuxer no longer owns a remuxer
- push now returns the following:
  - Synchronously returns a TransmuxerResult object when the segment data is unencrypted
  - Asynchronously returns a TransmuxerResult promise when the segment data is encrypted
Refactor demuxer-worker to transmuxer-worker
- Posts a transmuxComplete message when transmuxing is complete
Refactor demuxer to transmuxer-interface
- Accepts a new onTransmuxComplete handler
  - Triggered when transmuxComplete is received from a worker
  - Waits for the transmux promise to resolve for non-worker contexts; or triggers the handle immediately if synchronous
  - No longer attaches frag and id to each event; instead, the interface passes along a TransmuxIdentifier object which is used post-transmux to regain the fragment reference
Refactor mp4demuxer and passthrough-remuxer
- startDTS offsetting has been shifted from the demuxer to the remuxer
- InitSegment and InitPTS generation has been shifted from the demuxer to the remuxer
Refactor stream-controller and audio-stream-controller:
- No longer listens for FRAG_PARSING_DATA, FRAG_PARSED, FRAG_PARSING_DATA events
  - But still emits them for compatibility
- Each event is handled privately within the controller
- Shifts fragment load abort logic to the top of the handleTransmuxComplete handler
  - This means that a fragment either no data is buffered, or the entire fragment is buffered; previously, each event handler had it's own abort check logic. This could lead to cases such as audio being buffered, but not video.
- Cleans up handlers by relying on new assumptions (e.g. the only data received is of its expected type)
- handleTransmuxComplete emits the FRAG_PARSING_USERDATA and FRAG_PARSING_METADATA events
Refactors transmuxing flow to allow for progressive transmuxing:
- Demuxers implement a flush method which is called when progressive downloading completes
  - Exposed to controllers via flush on the transmuxer-worker
  - Clears any remaining data from the demuxer & calls remux once more
- Progressive loading is in the project but not yet implemented in controllers; flush is called after
  the load promise resolves
- Demuxers which do not support progressive parsing extend the new NonProgressiveDemuxer class, which buffers data until flush is called for the final remux
  - Currently all demuxers extend this class

Why is this Pull Request needed?

The event-driven architecture of the tranmuxing pipeline is not compatibly with a clean progressive segment parsing pipeline. By shifting to a return value based architecture, we're able to scope transmuxing to the controllers and segments to which the operation belongs. With this new architecture we will be able to safely transmux segments as a series of chunks, a core requirement for LHLS.

Are there any points in the code the reviewer needs to double check?

Yes

Resolves issues:

JW8-2638

tests/unit/utils/mp4-tools.js

src/demux/mp4demuxer.ts

src/demux/transmuxer.ts

robwalch · 2019-03-19T18:48:30Z

@johnBartos I got a TypeScript error:

$ yarn type-check                                          
yarn run v1.13.0
$ tsc --noEmit
src/controller/buffer-controller.ts:30:7 - error TS2415: Class 'BufferController' incorrectly extends base class 'EventHandler'.
  Property 'hls' is private in type 'BufferController' but not in type 'EventHandler'.

30 class BufferController extends EventHandler {
         ~~~~~~~~~~~~~~~~


Found 1 error.

jwplayer-robot · 2019-03-20T13:25:48Z

❗ Build for commit 64728e7 did not complete.
🏗️ jwplayer/hls.js build 67 FAILURE
🏗️ jwplayer-commercial build ${TRIGGERED_BUILD_NUMBER_jw7_commercial} ${TRIGGERED_BUILD_RESULT_jw7_commercial}
🥒 Allure Report ${TRIGGERED_BUILD_RESULT_jw7_commercial_cucumber}
🍆 Squash PR build ${TRIGGERED_BUILD_NUMBER_jw7_commercial}
🍆 Squash HLSJS Suite ${TRIGGERED_BUILD_NUMBER_jw7_commercial}

…t support progressive demuxing

…ing lastEndDTS to duration if already defined

…io-stream-controller

…tream controller

jwplayer-robot · 2019-03-20T18:21:00Z

❗ Build for commit 0b0131b did not complete.
🏗️ jwplayer/hls.js build 68 FAILURE
🏗️ jwplayer-commercial build ${TRIGGERED_BUILD_NUMBER_jw7_commercial} ${TRIGGERED_BUILD_RESULT_jw7_commercial}
🥒 Allure Report ${TRIGGERED_BUILD_RESULT_jw7_commercial_cucumber}
🍆 Squash PR build ${TRIGGERED_BUILD_NUMBER_jw7_commercial}
🍆 Squash HLSJS Suite ${TRIGGERED_BUILD_NUMBER_jw7_commercial}

jwplayer-robot · 2019-03-20T18:51:47Z

⚠️ Build for commit 61c5744 is unstable.
🏗️ jwplayer/hls.js build 69 SUCCESS
🏗️ jwplayer-commercial build 20544 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20544
🍆 Squash HLSJS Suite 20544

robwalch

Found a few failures:

Overall much better than before, but with a refactor this size you need to keep track of what is breaking. It may become more difficult to address these issues without compromising your refactor's new design the further you go.

robwalch · 2019-03-20T19:15:37Z

src/demux/mp4demuxer.ts

+  destroy () {}
+}
+
+const dummyTrack = () => ({ type: '', id: -1, pid: -1, inputTimeScale: 90000, sequenceNumber: -1, len: 0, samples: [] });


Maybe this should just be exported/imported from non-progressive-demuxer.ts or defined as it's own module/type?

jwplayer-robot · 2019-03-20T22:13:44Z

⚠️ Build for commit 3192fff is unstable.
🏗️ jwplayer/hls.js build 71 SUCCESS
🏗️ jwplayer-commercial build 20555 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20555
🍆 Squash HLSJS Suite 20555

johnBartos · 2019-03-21T15:47:27Z

Player does not resume from live pause http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/public/events/?config=hlsjs/hls-live-bloomberg.json
this.fragCurrent is null http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/public/events/?config=captions/captions-hls-embedded-vtt-korean.json
data is undefined http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/harness.html?automated=true&edition=ads&config=hlsjs%2Fhls-cbc-ima-midroll.json&setup=1
Exceptions thrown from handleTransmuxComplete: http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/harness.html?automated=true&edition=enterprise&config=hlsjs%2Fhls-archived-fox-dvr-force-hlsjs.json&setup=1 & http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/public/events/?config=hlsjs/hls-custom-labels-force-hlsjs.json & http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/harness.html?automated=true&edition=enterprise&config=hlsjs%2Fhls-encoding-uplynk-force-hlsjs.json&setup=1
Perhaps a stream issue, but this DVR stream is not starting: http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/public/events/?config=hlsjs/hls-dvr-single-bitrate.json
Stream plays but gap seeking behavior at the end of the stream has changed: http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/harness.html?automated=true&edition=ads&config=hlsjs%2Fhls-stream-with-large-gap-at-start-of-stream-autostart.json&setup=1
CORS error code is different: http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/harness.html?automated=true&edition=ads&config=hlsjs%2Fhls-cors-bunny.json&setup=1
Stream does not start; possible backtracking issue: http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/harness.html?automated=true&edition=ads&config=hlsjs%2Fhls-dvr-hockey-archive.json&setup=1
Possible ID3 exceptions, but haven't been able to repro locally: http://player-pr-test-jenkins.longtailvideo.com/builds/20555/archive/test/harness.html?setup&edition=ads&language=&config=hlsjs/hls-id3-timed-metadata&feature=hlsjs/id3_timed_metadata_and_program_date_time_in_manifest_using_hlsjs_provider

Dead stream link causing failures: https://bob.jwplayer.com/~alex/121598/new_master.m3u8

jwplayer-robot · 2019-03-22T18:10:48Z

⚠️ Build for commit 8479379 is unstable.
🏗️ jwplayer/hls.js build 72 SUCCESS
🏗️ jwplayer-commercial build 20571 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20571
🍆 Squash HLSJS Suite 20571

jwplayer-robot · 2019-03-22T19:13:29Z

⚠️ Build for commit a357b0f is unstable.
🏗️ jwplayer/hls.js build 73 SUCCESS
🏗️ jwplayer-commercial build 20573 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20573
🍆 Squash HLSJS Suite 20573

jwplayer-robot · 2019-03-22T20:30:44Z

⚠️ Build for commit bb2ceb4 is unstable.
🏗️ jwplayer/hls.js build 74 SUCCESS
🏗️ jwplayer-commercial build 20581 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20581
🍆 Squash HLSJS Suite 20581

johnBartos · 2019-03-22T21:52:23Z

Latest bugs:

Duration is very large (and incorrect) http://player-pr-test-jenkins.longtailvideo.com/builds/20581/archive/test/harness.html?automated=true&edition=enterprise&config=captions%2Fcaptions-hls-embedded-vtt-korean.json&setup=1
AES token erroring on play http://player-pr-test-jenkins.longtailvideo.com/builds/20581/archive/test/harness.html?automated=true&edition=enterprise&config=hlsjs%2Fhls-aestoken-sintel-force-hlsjs.json&setup=1
JS AES decrypt is broken (use any AES-128 stream) //playertest.longtailvideo.com/hls/fox-single-quality/g-nodisco.m3u8

jwplayer-robot · 2019-03-22T22:26:15Z

⚠️ Build for commit 6b5cea5 is unstable.
🏗️ jwplayer/hls.js build 75 SUCCESS
🏗️ jwplayer-commercial build 20585 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20585
🍆 Squash HLSJS Suite 20585

jwplayer-robot · 2019-03-25T16:03:24Z

⚠️ Build for commit 17e70c6 is unstable.
🏗️ jwplayer/hls.js build 76 SUCCESS
🏗️ jwplayer-commercial build 20587 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20587
🍆 Squash HLSJS Suite 20587

jwplayer-robot · 2019-03-25T17:46:55Z

⚠️ Build for commit 13ccd6a is unstable.
🏗️ jwplayer/hls.js build 79 SUCCESS
🏗️ jwplayer-commercial build 20592 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20592
🍆 Squash HLSJS Suite 20592

johnBartos · 2019-03-25T18:10:00Z

TODO: Pass AES token into provider fetch config

jwplayer-robot · 2019-03-25T18:44:52Z

⚠️ Build for commit 6f2ef77 is unstable.
🏗️ jwplayer/hls.js build 82 SUCCESS
🏗️ jwplayer-commercial build 20594 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20594
🍆 Squash HLSJS Suite 20594

…ization

jwplayer-robot · 2019-03-25T20:06:48Z

⚠️ Build for commit f315427 is unstable.
🏗️ jwplayer/hls.js build 83 SUCCESS
🏗️ jwplayer-commercial build 20597 SUCCESS
🥒 Allure Report UNSTABLE
🍆 Squash PR build 20597
🍆 Squash HLSJS Suite 20597

johnBartos requested a review from robwalch February 25, 2019 20:45

johnBartos added the enhancement label Feb 25, 2019

johnBartos force-pushed the feature/demuxing-refactor branch 2 times, most recently from fa04412 to d725cd0 Compare February 27, 2019 21:36

johnBartos changed the base branch from feature/frag-loading-refactor to LHLS February 27, 2019 21:38

johnBartos force-pushed the feature/demuxing-refactor branch 3 times, most recently from b80d6a7 to 988f82a Compare February 27, 2019 22:54

Refactor transmuxing workflow

64728e7

johnBartos force-pushed the feature/demuxing-refactor branch from a6b9237 to 64728e7 Compare March 13, 2019 18:41

robwalch reviewed Mar 18, 2019

View reviewed changes

tests/unit/utils/mp4-tools.js Outdated Show resolved Hide resolved

robwalch suggested changes Mar 18, 2019

View reviewed changes

src/demux/mp4demuxer.ts Outdated Show resolved Hide resolved

src/demux/transmuxer.ts Outdated Show resolved Hide resolved

John Bartos added 6 commits March 20, 2019 11:47

Create NonProgressiveDemuxer as a base class for demuxers which do no…

599016b

…t support progressive demuxing

Implement flush in non-worker context

65dc7ee

Refactor flush interface

e36a6cc

Implement cmaf sample counting for duration computation, avoid assign…

ed76300

…ing lastEndDTS to duration if already defined

Implement NonProgressiveDemuxer for TS Demuxing; wire up flush in aud…

def0281

…io-stream-controller

Extend all demuxers with NonProgressiveDemuxer; fix errors in audio/s…

0b0131b

…tream controller

Address feedback

61c5744

johnBartos requested a review from robwalch March 20, 2019 18:36

robwalch approved these changes Mar 20, 2019

View reviewed changes

John Bartos added 2 commits March 20, 2019 17:42

Address transmuxing bugs

ff9c1fe

Implement shared dummyTrack function

3192fff

Implement caching for demuxer probing

ba414c8

John Bartos added 5 commits March 21, 2019 17:29

Ensure that demuxer tracks always exist

21a04a8

Handle AES-128 decryption with flush

1477062

Fix gaps caused by erroneous demuxer reset

fcacb43

Fix backtracking stall during live streams

d2c9ce9

Remove console logs

8479379

Clear text & id3 samples after remuxing

a357b0f

Fix reference to this.demuxer in stream-controller

bb2ceb4

Pass in fragment data as an ArrayBuffer to the decrypter

6b5cea5

Ensure that the initSegment is reset upon muxer creation

17e70c6

Reset nextLoadPosition of the fragment load was aborted

13ccd6a

Remove logs

6f2ef77

Ensure that initSegment data is passed through during remuxer initial…

f315427

…ization

johnBartos merged commit 6836ef3 into LHLS Mar 26, 2019

johnBartos mentioned this pull request Sep 13, 2019

v1.0.0 (LHLS Chunked Transfer Support, IMSC1, Refactor) video-dev/hls.js#2370

Closed

robwalch deleted the feature/demuxing-refactor branch January 27, 2020 17:53

robwalch mentioned this pull request Oct 5, 2020

v1.0.0: HLS v10 Support and Transmuxing Refactor video-dev/hls.js#3072

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor transmuxing data flow #199

Refactor transmuxing data flow #199

johnBartos commented Feb 25, 2019 •

edited

robwalch commented Mar 19, 2019 •

edited

jwplayer-robot commented Mar 20, 2019

jwplayer-robot commented Mar 20, 2019

jwplayer-robot commented Mar 20, 2019

robwalch left a comment

robwalch Mar 20, 2019

johnBartos Mar 20, 2019

jwplayer-robot commented Mar 20, 2019

johnBartos commented Mar 21, 2019 •

edited

jwplayer-robot commented Mar 22, 2019

jwplayer-robot commented Mar 22, 2019

jwplayer-robot commented Mar 22, 2019

johnBartos commented Mar 22, 2019

jwplayer-robot commented Mar 22, 2019

jwplayer-robot commented Mar 25, 2019

jwplayer-robot commented Mar 25, 2019

johnBartos commented Mar 25, 2019

jwplayer-robot commented Mar 25, 2019

jwplayer-robot commented Mar 25, 2019

Refactor transmuxing data flow #199

Refactor transmuxing data flow #199

Conversation

johnBartos commented Feb 25, 2019 • edited

This PR will...

Why is this Pull Request needed?

Are there any points in the code the reviewer needs to double check?

Resolves issues:

robwalch commented Mar 19, 2019 • edited

jwplayer-robot commented Mar 20, 2019

jwplayer-robot commented Mar 20, 2019

jwplayer-robot commented Mar 20, 2019

robwalch left a comment

Choose a reason for hiding this comment

robwalch Mar 20, 2019

Choose a reason for hiding this comment

johnBartos Mar 20, 2019

Choose a reason for hiding this comment

jwplayer-robot commented Mar 20, 2019

johnBartos commented Mar 21, 2019 • edited

jwplayer-robot commented Mar 22, 2019

jwplayer-robot commented Mar 22, 2019

jwplayer-robot commented Mar 22, 2019

johnBartos commented Mar 22, 2019

jwplayer-robot commented Mar 22, 2019

jwplayer-robot commented Mar 25, 2019

jwplayer-robot commented Mar 25, 2019

johnBartos commented Mar 25, 2019

jwplayer-robot commented Mar 25, 2019

jwplayer-robot commented Mar 25, 2019

johnBartos commented Feb 25, 2019 •

edited

robwalch commented Mar 19, 2019 •

edited

johnBartos commented Mar 21, 2019 •

edited