Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fast-forward implementation v2: virtual dom optimization #853

Merged
merged 91 commits into from
May 10, 2022

Conversation

YunFeng0817
Copy link
Member

Motivation

In PR #241, we applied a virtual parent optimization to the fast-forward playback where we got a 10 times performance improvement. But this approach still has some unavoidable defects, e.g. #550, #529, #827.

The cause and solutions of these issues have been explained here: #548 (comment). I believe the virtual dom could be a good solution to solve these issues and it can also bring an approximately 2.5x performance boost.

changes

This PR has a huge amount of code changes and I'll explain them to help understand.

  1. In rrdom: an ad-hoc DOM for rrweb session data #419 and PR rrdom #613, we designed and implemented an ad-hoc DOM called 'rrdom' as an environment to replay the session. In this PR, I make some changes based on it and make it a browser runtime specially designed for virtual-dom. The feature of this runtime is that it is slim and only covers the dom operations used in the rrweb replayer. In other words, the virtual dom is implemented with rrdom.
  2. The main mechanism of virtual dom optimization is described below. At the start of fast-forward process, a virtual dom tree is created which is a clone of the real dom tree. More precisely, the virtual dom is created right before the replayer starts to process the first node mutation event. After that, any node mutations are applied to the virtual dom tree. After all incremental events have been applied synchronously, the replayer goes into the flush stage. At this stage, the replayer executes a diff algorithm between the virtual-dom and the real dom, and applies all mutations to the real dom.
  3. PR fast-forward implementation v1 #241 uses a data structure called treeIndex to improve the performance of applying input events and scroll events. This PR takes a similar approach to replace it. The key idea is to delay the handling of these types of events until the flush stage. Input events, scroll events, canvas mutations, and CSS style mutations are stored on their target virtual nodes at first. These data will be processed only after the real dom is updated in the flush stage. BTW, the virtual stylesheet created by Buffer modifications to virtual stylesheets #618 is also integrated into the virtual dom in this PR.
  4. To improve the quality and reliability of this PR, the rrdom package has reached a high test coverage listed below:
File % Stmts % Branch % Funcs % Lines
All files 97.35 94.56 97.28 97.35
diff.ts 92.34 91.33 91.66 92.15
document-nodejs.ts 100 100 100 100
document.ts 100 100 100 100
polyfill.ts 95.83 100 90.9 95.34
style.ts 100 83.33 100 100
virtual-dom.ts 97.6 91.78 89.28 98.37

Several integration test cases are also added to cover some types of replay scenes.

Some minor changes may relate to these PRs: #427, #481, #662, #666, #671, #756.

cssom version has to be above 0.5.0 to pass virtual style tests
Cause the attribute viewBox is case sensitive, set value for viewbox doesn't work
Error: @rollup/plugin-typescript TS1371: This import is never used as a value and must use 'import type' because the 'importsNotUsedAsValues' is set to 'error'
@YunFeng0817 YunFeng0817 merged commit 56abfc5 into master May 10, 2022
Yuyz0112 pushed a commit that referenced this pull request May 12, 2022
…tion (#895)

* rrdom: add a diff function for properties

* implement diffChildren function and unit tests

* finish basic functions of diff algorithm

* fix several bugs in the diff algorithm

* replace the virtual parent optimization in applyMutation()

* fix: moveAndHover after the diff algorithm is executed

* replace virtual style map with rrdom

cssom version has to be above 0.5.0 to pass virtual style tests

* fix: failed virtual style tests in replayer.test.ts

* fix: failed polyfill tests caused by nodejs compatibility of different versions

* fix: svg viewBox attribute doesn't work

Cause the attribute viewBox is case sensitive, set value for viewbox doesn't work

* feat: replace treeIndex optimization with rrdom

* fix bug of diffProps and disable smooth scrolling animation in fast-forward mode

* feat: add iframe support

* fix: @rollup/plugin-typescript build errors in rrweb-player

Error: @rollup/plugin-typescript TS1371: This import is never used as a value and must use 'import type' because the 'importsNotUsedAsValues' is set to 'error'

* fix: bug when fast-forward input events and add test for it

* add test for fast-forward scroll events

* fix: custom style rules don't get inserted into some iframe elements

* code style tweak

* fix: enable to diff iframe elements

* fix  the jest error "Unexpected token 'export'"

* try to fix build error of rrweb-player

* correct the attributes definition in rrdom

* fix: custom style rules are not inserted in some iframes

* add support for shadow dom

* add support for MediaInteraction

* add canvas support

* fix unit test error in rrdom

* add support for Text, Comment

* try to refactor RRDom

* refactor RRDom to reduce duplicate code

* rename document-browser to virtual-dom

* increase the test coverage for document.ts and add ownerDocument for it

* Merge branch 'master' into virtual-dom

* add more test for virtual-dom.ts

* use cssstyle in document-nodejs

* fix: bundle error

* improve document-nodejs

* enable to diff scroll positions of an element

* rename rrdom to virtualDom for more readability and make the tree public

* revert unknown change

* improve the css style parser for comments

* improve code style

* update typings

* add handling for the case where legacy_missingNodeRetryMap is not empty

* only import types from rrweb into rrdom

* Apply suggestions from code review

Co-authored-by: Justin Halsall <Juice10@users.noreply.github.com>

* Apply suggestions from code review

* fix building error in rrweb

* add a method setDefaultSN to set a default value for a RRNode's __sn

* fix rrweb test error and bump up other packages

* add support for custom property of css styles

* add a switch for virtual-dom optimization

* Apply suggestions from code review

1. add an enum type for NodeType
2. rename nodeType from rrweb-snapshot to RRNodeType
3. rename notSerializedId to unserializedId
4. add comments for some confusing variables

* adapt changes of #865 to virtual-dom and improve the test case for more coverage

* apply review suggestions

#853 (review)

* tweak the diff algorithm

* add description of the flag useVirtualDom and remove outdated logConfig

* Remove console.log

* Contain changes to document

* Upgrade rollup to 2.70.2

* Revert "Upgrade rollup to 2.70.2"

This reverts commit b1be81a.

* Fix type checking rrdom

* Fix typing error while bundling

* Fix tslib error on build

Rollup would output the following error:
`semantic error TS2343: This syntax requires an imported helper named '__spreadArray' which does not exist in 'tslib'. Consider upgrading your version of 'tslib'.`

* Increase memory limit for rollup

* Use esbuild for bundling

Speeds up bundling significantly

* Avoid circular dependencies and import un-bundled rrdom

* Fix imports

* Revert back to pre-esbuild

This reverts the following commits:
b7b3c8d
72e23b8
85d600a
61e1a5d

* Set node to lts (12 is no longer supported)

* Speed up bundling and use less memory

This fixes the out of memory errors happening while bundling

* remove __sn from rrdom

* fix typo

* test: add a test case for StyleSheet mutation exceptions while fast-forwarding

* rename Array.prototype.slice.call() to Array.from()

* improve test cases

* fix: PR #887 in 'virtual-dom' branch

* apply justin's suggestion on 'Array.from' refactor

related commit 0f6729d

* improve import code structure

Co-authored-by: Yun Feng <yun.feng@anu.edu.au>
Vadman97 pushed a commit to highlight/rrweb that referenced this pull request Jun 2, 2022
* implement diffChildren function and unit tests

* finish basic functions of diff algorithm

* fix several bugs in the diff algorithm

* replace the virtual parent optimization in applyMutation()

* fix: moveAndHover after the diff algorithm is executed

* replace virtual style map with rrdom

cssom version has to be above 0.5.0 to pass virtual style tests

* fix: failed virtual style tests in replayer.test.ts

* fix: failed polyfill tests caused by nodejs compatibility of different versions

* fix: svg viewBox attribute doesn't work

Cause the attribute viewBox is case sensitive, set value for viewbox doesn't work

* feat: replace treeIndex optimization with rrdom

* fix bug of diffProps and disable smooth scrolling animation in fast-forward mode

* feat: add iframe support

* fix: @rollup/plugin-typescript build errors in rrweb-player

Error: @rollup/plugin-typescript TS1371: This import is never used as a value and must use 'import type' because the 'importsNotUsedAsValues' is set to 'error'

* fix: bug when fast-forward input events and add test for it

* add test for fast-forward scroll events

* fix: custom style rules don't get inserted into some iframe elements

* code style tweak

* fix: enable to diff iframe elements

* fix  the jest error "Unexpected token 'export'"

* try to fix build error of rrweb-player

* correct the attributes definition in rrdom

* fix: custom style rules are not inserted in some iframes

* add support for shadow dom

* add support for MediaInteraction

* add canvas support

* fix unit test error in rrdom

* add support for Text, Comment

* try to refactor RRDom

* refactor RRDom to reduce duplicate code

* rename document-browser to virtual-dom

* increase the test coverage for document.ts and add ownerDocument for it

* Merge branch 'master' into virtual-dom

* add more test for virtual-dom.ts

* use cssstyle in document-nodejs

* fix: bundle error

* improve document-nodejs

* enable to diff scroll positions of an element

* rename rrdom to virtualDom for more readability and make the tree public

* revert unknown change

* improve the css style parser for comments

* improve code style

* update typings

* add handling for the case where legacy_missingNodeRetryMap is not empty

* only import types from rrweb into rrdom

* Apply suggestions from code review

Co-authored-by: Justin Halsall <Juice10@users.noreply.github.com>

* Apply suggestions from code review

* fix building error in rrweb

* add a method setDefaultSN to set a default value for a RRNode's __sn

* fix rrweb test error and bump up other packages

* add support for custom property of css styles

* add a switch for virtual-dom optimization

* Apply suggestions from code review

1. add an enum type for NodeType
2. rename nodeType from rrweb-snapshot to RRNodeType
3. rename notSerializedId to unserializedId
4. add comments for some confusing variables

* adapt changes of #865 to virtual-dom and improve the test case for more coverage

* apply review suggestions

rrweb-io/rrweb#853 (review)

* tweak the diff algorithm

* add description of the flag useVirtualDom and remove outdated logConfig

* Remove console.log

* Contain changes to document

* Upgrade rollup to 2.70.2

* Revert "Upgrade rollup to 2.70.2"

This reverts commit b1be81a2a76565935c9dc391f31beb7f64d25956.

* Fix type checking rrdom

* Fix typing error while bundling

* Fix tslib error on build

Rollup would output the following error:
`semantic error TS2343: This syntax requires an imported helper named '__spreadArray' which does not exist in 'tslib'. Consider upgrading your version of 'tslib'.`

* Increase memory limit for rollup

* Use esbuild for bundling

Speeds up bundling significantly

* Avoid circular dependencies and import un-bundled rrdom

* Fix imports

* Revert back to pre-esbuild

This reverts the following commits:
b7b3c8dbaa551a0129da1477136b1baaad28e6e1
72e23b8e27f9030d911358d3a17fe5ad1b3b5d4f
85d600a20c56cfa764cf1f858932ba14e67b1d23
61e1a5d323212ca8fbe0569e0b3062ddd53fc612

* Set node to lts (12 is no longer supported)

* Speed up bundling and use less memory

This fixes the out of memory errors happening while bundling

* remove __sn from rrdom

* fix typo

* test: add a test case for StyleSheet mutation exceptions while fast-forwarding

* rename Array.prototype.slice.call() to Array.from()

* improve test cases

* fix: PR #887 in 'virtual-dom' branch

* apply justin's suggestion on 'Array.from' refactor

related commit 0f6729d27a323260b36fbe79485a86715c0bc98a

* improve import code structure

Co-authored-by: Yun Feng <yun.feng@anu.edu.au>
@YunFeng0817 YunFeng0817 deleted the virtual-dom branch July 11, 2022 06:14
@Juice10 Juice10 added the 2.0 label Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants