Bundler Refactor #8170

kumavis · 2020-03-07T11:04:24Z

status: ready for additional feedback or merge 🎸

refactor of the bundler using lavamoat-friendly bundle factoring. this does not add lavamoat to the bundles.

changes to bundle result

the ui-libs and bg-libs have been replaced by the common bundle. common contains all packages that are used by both the background and the ui. the "factoring" into the common bundle is automatic and does not need any adjustment as modules are added/removed.

here is a breakdown of the resultant bundle sizes under different minification strategies. the current configuration is the strategy all the way to the right. compare the size of bg/ui/common.

changes to bundling process

I tried to refactor the bundler so that it was easier to follow without having too much repeated code, but im not sure that I succeeded. part of what complicates this is that certain things (e.g. sourcemaps) need to configure both:

initial bundle options (enable browserify debug option) and
post-bundle build pipeline (load + write sourcemaps)

its additionally complicated because a watchify re-build involves creating a new build pipeline. my proposed solution is as follows

a basic bundle+build config obj

{ bundlerOpts: {}, events: new EventEmitter() }

events will fire a 'pipeline' event when the build pipeline is ready to be augmented. The pipeline is based on browserify's own internal pipeline, a labeled-stream-splicer. It is essentially a nested array of streams that you can pop into position, and it handles all the plumbing for you. it may be slightly over-engineered, but it allows our very different build configurations (inpage+contentscript standalone, ui+background factor) share the same interface, and makes applying some feature (e.g. sourcemaps) able to happen in one place

etc

moved the lavamoat dashboard (sesify-viz) to a separate process that runs parallel with the other builds, should speed up CI ~3min

metamaskbot · 2020-03-08T12:00:12Z

Builds ready [915c7c8]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
dep viz: background
all artifacts

Page Load Metrics (629 ± 29 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	34	135	47	22	10
		domContentLoaded	456	739	627	60	29
		load	457	740	629	60	29
		domInteractive	455	739	627	60	29

metamaskbot · 2020-03-09T01:23:02Z

Builds ready [aaedd5f]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
dep viz: background
all artifacts

Page Load Metrics (621 ± 41 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	36	52	40	4	2
		domContentLoaded	435	773	620	85	41
		load	436	775	621	85	41
		domInteractive	434	773	620	85	41

metamaskbot · 2020-03-09T04:12:21Z

Builds ready [b43fc43]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
dep viz: background
all artifacts

Page Load Metrics (590 ± 43 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	34	46	37	3	1
		domContentLoaded	290	652	588	90	43
		load	293	654	590	90	43
		domInteractive	290	651	588	90	43

metamaskbot · 2020-03-09T05:32:22Z

Builds ready [ce9c9aa]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, common
dep viz: ui + background
all artifacts

Page Load Metrics (688 ± 31 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	34	189	53	40	19
		domContentLoaded	607	844	686	64	31
		load	609	847	688	65	31
		domInteractive	607	844	686	64	31

metamaskbot · 2020-03-09T07:22:00Z

Builds ready [89a1802]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, common
dep viz: ui + background
all artifacts

Page Load Metrics (871 ± 92 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	42	184	82	47	23
		domContentLoaded	352	1416	868	191	92
		load	353	1420	871	192	92
		domInteractive	351	1415	868	191	92

metamaskbot · 2020-03-09T08:11:15Z

Builds ready [f8b0107]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, common
dep viz: ui + background
all artifacts

Page Load Metrics (703 ± 67 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	35	140	53	25	12
		domContentLoaded	447	1053	700	139	67
		load	455	1055	703	139	67
		domInteractive	446	1053	699	139	67

metamaskbot · 2020-03-13T09:44:03Z

Builds ready [fa96846]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, common
dep viz: ui + background
all artifacts

Page Load Metrics (646 ± 11 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	35	54	39	4	2
		domContentLoaded	612	699	644	24	11
		load	614	701	646	24	11
		domInteractive	612	698	644	24	11

* origin/develop: (582 commits) Use async/await for seedPhraseVerifier.verifyAccounts (#9100) Use async/await for getRestrictedMethods (#9099) Update dependencies (#9105) update email us to contact us (#9104) Improve source maps (#9101) Update font family globally (#9073) rpc-cap@3.1.0 (#9103) Use environment variable for production Sentry DSN (#9097) Only log error on first occurrence of missing substitution (#9096) Use mixins for typography instead of placeholder selectors (#9072) Update css folder structure (#9071) Disable Sentry in development (#9095) Use environment variable for MetaMetrics project ID (#9094) Use development metametrics project during tests (#9093) json-rpc-engine@5.2.0 (#9091) fixup! call initializeProvider where necessary call initializeProvider where necessary Add euclid fontface (#9018) fix timing-reliant network controller test Robustify permissions controller requestUserApproval tests (#9064) ...

metamaskbot · 2020-07-30T15:53:54Z

Builds ready [1e042f2]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, common, phishing-detect
dep viz: ui + background
all artifacts

Page Load Metrics (502 ± 40 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	28	122	40	19	9
		domContentLoaded	353	646	500	83	40
		load	354	647	502	83	40
		domInteractive	352	645	500	83	40

Gudahtt · 2020-07-30T16:26:22Z

The bundles produced by this branch look good, but the sourcemaps for common.js are totally wrong 🤔

I'll try and reduce the size of this PR first by getting a few of these changes merged separately, then I'll investigate that problem.

Edit: Done. This PR is as small as it'll get for now. It might still be able to be split between the build changes and the new lavamoat build step; TBD.

There are no functional changes. This was extracted from #8170

* origin/develop: Factor out `getEnvironment` function in build script (#9114) Update `browserify` from v16.2.3 to v16.5.1 (#9113) Update `sesify-viz` from v3.0.9 to v3.0.10 (#9111) Update `gulp-rename` from v1.4.0 to v2.0.0 (#9112) Update `source-map-explorer` from v2.0.1 to v2.4.2 (#9110) Add `build-artifacts` to .gitignore (#9109)

`browserify-transform-tools` is now used indirectly via `lavamoat-browserify`, which has also replaced `sesify`.

metamaskbot · 2020-07-30T20:03:14Z

Builds ready [4b77117]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, common, phishing-detect
dep viz: ui + background
all artifacts

Page Load Metrics (514 ± 45 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	27	135	40	22	11
		domContentLoaded	403	679	512	95	45
		load	404	680	514	95	45
		domInteractive	403	678	511	95	45

* origin/develop: Update `brfs` from v1.6.1 to v2.0.2 (#9115)

Gudahtt · 2020-07-30T21:14:46Z

It looks like the source maps for all of the bundles are messed up. The sources entry in ui.js.map lists a bunch of background modules, for example.

Edit: It turns out that just the three "factored" bundles are affected (ui.js, background.js, and common.js).

metamaskbot · 2020-07-30T21:18:35Z

Builds ready [8b8bce8]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, common, phishing-detect
dep viz: ui + background
all artifacts

Page Load Metrics (421 ± 52 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	26	40	32	3	2
		domContentLoaded	253	636	420	108	52
		load	255	638	421	108	52
		domInteractive	253	636	419	108	52

Gudahtt · 2020-08-04T14:15:05Z

I tried looking at the direct output of bify-package-factor to check the inline source maps, and they seem correct 🤔 Or at least they weren't as obviously incorrect as the final output. So maybe the problem here is with something we're doing after bify-package-factor.

* origin/develop: Fix connection removal bug (#9137) Add source map validator to CI (#9135) Update source map validator target files (#9133) Improve sourcemap validator console report (#9131) Add `validate-source-maps` npm script (#9134) Non-zero exit code upon failure to validate source maps (#9132) remove unused tx-list styles (#9121) delete unused confirm styles (#9118)

Gudahtt · 2020-08-05T02:21:32Z

I've just merged develop into this again to bring in the source map validator script changes (and I updated it to check the common.js bundle), so CI should now fail until the source maps are fixed.

There are no functional changes. This was extracted from #8170

kumavis · 2021-01-18T09:05:39Z

this PR is super stale now, but has some potentially useful stuff

kumavis · 2021-01-18T09:07:13Z

i want to revert this to a draft pr but i dont see an option for that?

brad-decker · 2021-01-18T15:55:03Z

I didn't know this existed until Whymarrh pointed it out to me. Paying it forward, also marking this as draft.

kumavis · 2021-04-27T23:50:11Z

rip

kumavis requested review from danjm, Gudahtt and whymarrh as code owners March 7, 2020 11:04

kumavis changed the base branch from develop to build-sys-1 March 7, 2020 13:37

kumavis changed the base branch from build-sys-1 to develop March 9, 2020 00:55

build/scripts - refactor to use lavamoat-friendly bundle factoring

aaedd5f

kumavis force-pushed the build-sys-2 branch from 915c7c8 to aaedd5f Compare March 9, 2020 01:03

kumavis added 2 commits March 9, 2020 11:41

build/scripts - add lavamoat config generation task

b43fc43

phishing-detect - add to background

82e6bd5

kumavis added 6 commits March 9, 2020 12:52

build/scripts - shorten output bundle names

000fc73

development - update source-map-explorer

45aec1f

development - update metamaskbot links

78bad65

manifest - update bundle location

ad73a61

build/scripts - rename bundler utility

0dc9efc

build/scripts - remove unused code

ce9c9aa

kumavis added 2 commits March 9, 2020 14:39

build/scripts - fix sourcemaps for factor bundle

26bcfea

deps - bump source-map-explorer

89a1802

build/scripts - breakout setupSourcemaps

f8b0107

build/scripts - clean and comment

c2475f7

This comment has been minimized.

Sign in to view

kumavis added 2 commits March 13, 2020 17:13

build/scripts - factor some of the default settings

7c7ec0f

build/scripts - bangarang

fa96846

kumavis mentioned this pull request Apr 20, 2020

Fix external sourcemaps reference causing error in browser console #8367

Merged

wbt mentioned this pull request May 12, 2020

"DevTools failed to parse SourceMap" #8167

Closed

kumavis requested a review from a team as a code owner July 30, 2020 15:32

Gudahtt added a commit that referenced this pull request Jul 30, 2020

Factor out getEnvironment function in build script

bce8340

There are no functional changes. This was extracted from #8170

Gudahtt mentioned this pull request Jul 30, 2020

Factor out getEnvironment function in build script #9114

Merged

Gudahtt added a commit that referenced this pull request Jul 30, 2020

Factor out getEnvironment function in build script (#9114)

9e12cdc

There are no functional changes. This was extracted from #8170

Gudahtt added 2 commits July 30, 2020 16:21

Remove unused dependencies

4b77117

`browserify-transform-tools` is now used indirectly via `lavamoat-browserify`, which has also replaced `sesify`.

Merge remote-tracking branch 'origin/develop' into build-sys-2

8b8bce8

* origin/develop: Update `brfs` from v1.6.1 to v2.0.2 (#9115)

Gudahtt added a commit that referenced this pull request Aug 7, 2020

Factor out getEnvironment function in build script (#9114)

d4c3fa6

There are no functional changes. This was extracted from #8170

Gudahtt mentioned this pull request Jan 5, 2021

Create separate entry point for a notification.js build #6644

Closed

2 tasks

build - dont include sourcemaps comment in prod (#8367)

5e47cf9

brad-decker marked this pull request as draft January 18, 2021 15:55

kumavis mentioned this pull request Mar 25, 2021

build - refactor build system for easier configuration #10718

Merged

kumavis closed this Apr 27, 2021

kumavis deleted the build-sys-2 branch April 27, 2021 23:50

github-actions bot locked and limited conversation to collaborators Apr 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundler Refactor #8170

Bundler Refactor #8170

kumavis commented Mar 7, 2020 •

edited

metamaskbot commented Mar 8, 2020

metamaskbot commented Mar 9, 2020

metamaskbot commented Mar 9, 2020

metamaskbot commented Mar 9, 2020

metamaskbot commented Mar 9, 2020

metamaskbot commented Mar 9, 2020

This comment has been minimized.

This comment has been minimized.

metamaskbot commented Mar 13, 2020

metamaskbot commented Jul 30, 2020

Gudahtt commented Jul 30, 2020 •

edited

metamaskbot commented Jul 30, 2020

Gudahtt commented Jul 30, 2020 •

edited

metamaskbot commented Jul 30, 2020

Gudahtt commented Aug 4, 2020

Gudahtt commented Aug 5, 2020

kumavis commented Jan 18, 2021

kumavis commented Jan 18, 2021

brad-decker commented Jan 18, 2021

kumavis commented Apr 27, 2021

Bundler Refactor #8170

Bundler Refactor #8170

Conversation

kumavis commented Mar 7, 2020 • edited

changes to bundle result

changes to bundling process

etc

metamaskbot commented Mar 8, 2020

metamaskbot commented Mar 9, 2020

metamaskbot commented Mar 9, 2020

metamaskbot commented Mar 9, 2020

metamaskbot commented Mar 9, 2020

metamaskbot commented Mar 9, 2020

This comment has been minimized.

This comment has been minimized.

metamaskbot commented Mar 13, 2020

metamaskbot commented Jul 30, 2020

Gudahtt commented Jul 30, 2020 • edited

metamaskbot commented Jul 30, 2020

Gudahtt commented Jul 30, 2020 • edited

metamaskbot commented Jul 30, 2020

Gudahtt commented Aug 4, 2020

Gudahtt commented Aug 5, 2020

kumavis commented Jan 18, 2021

kumavis commented Jan 18, 2021

brad-decker commented Jan 18, 2021

kumavis commented Apr 27, 2021

kumavis commented Mar 7, 2020 •

edited

Gudahtt commented Jul 30, 2020 •

edited

Gudahtt commented Jul 30, 2020 •

edited