Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom faster ESM to CommonJS transformer #4366

Merged
merged 24 commits into from
Dec 18, 2020
Merged

Custom faster ESM to CommonJS transformer #4366

merged 24 commits into from
Dec 18, 2020

Conversation

devongovett
Copy link
Member

Currently, when not scope hoisting, we compile ES modules to CommonJS using @babel/plugin-transform-modules-commonjs. I noticed while benchmarking that this appeared to be quite slow - by far the slowest thing we do in Parcel. Looking through the source code, it appears to do multiple traversals over the entire AST. I'm sure there's a reason for that, but I don't think we need to do it for our use in Parcel. In addition, using @babel/core has some overhead over just using @babel/traverse directly.

This PR implements a new AST visitor that transforms ESM to CommonJS in a single pass over only the top-level statements in the program. Using scope information collected by babel traverse already, we can replace references without doing multiple full program traverses. On the esbuild benchmark, this improved overall build performance by ~25%!

In addition, builds may be slightly smaller because the helpers for various interop functionality are not duplicated in every file but included once in a separate asset.

@height
Copy link

height bot commented Mar 22, 2020

Link Height tasks by mentioning a task ID in the pull request title or description, commit messages, or comments.

You can also use "Close T-X" to automatically close a task when the pull request is merged.

@parcel-benchmark
Copy link

parcel-benchmark commented Mar 23, 2020

Benchmark Results

Kitchen Sink ✅

Timings

Description Time Difference
Cold 5.60s -195.00ms
Cached 856.00ms +43.00ms ⚠️

Cold Bundles

Bundle Size Difference Time Difference
dist/legacy/parcel.d5807e82.webp 102.94kb +0.00b 39.00ms +9.00ms ⚠️
dist/modern/parcel.d5807e82.webp 102.94kb +0.00b 69.00ms +5.00ms ⚠️
dist/modern/index.df30fe40.js 1.07kb +0.00b 1.31s +131.00ms ⚠️
dist/modern/index.html 701.00b +0.00b 1.43s +103.00ms ⚠️

Cached Bundles

Bundle Size Difference Time Difference
dist/legacy/parcel.d5807e82.webp 102.94kb +0.00b 82.00ms +21.00ms ⚠️
dist/modern/parcel.d5807e82.webp 102.94kb +0.00b 54.00ms -7.00ms 🚀
dist/modern/index.df30fe40.js 1.07kb +0.00b 92.00ms +12.00ms ⚠️

React HackerNews ✅

Timings

Description Time Difference
Cold 24.64s +353.00ms
Cached 12.90s -32.00ms

Cold Bundles

Bundle Size Difference Time Difference
dist/NotFound.141e0b29.js 532.00b +0.00b 277.00ms +21.00ms ⚠️
dist/logo.24c8bf9e.png 274.00b +0.00b 32.00ms -13.00ms 🚀

Cached Bundles

Bundle Size Difference Time Difference
dist/index.js 479.93kb +0.00b 56.00ms -10.00ms 🚀
dist/PermalinkedComment.a5120a24.js 4.22kb +0.00b 56.00ms -10.00ms 🚀
dist/UserProfile.95e14af2.js 1.70kb +0.00b 56.00ms -10.00ms 🚀
dist/NotFound.141e0b29.js 532.00b +0.00b 39.00ms -16.00ms 🚀

AtlasKit Editor ✅

Timings

Description Time Difference
Cold 2.26m -1.25s
Cached 3.10s -2.00ms

Cold Bundles

Bundle Size Difference Time Difference
dist/media-viewer.52bfe015.js 72.71kb +0.00b 13.49s +1.64s ⚠️
dist/card.93c390cd.js 53.76kb +0.00b 10.98s -1.06m 🚀
dist/Modal.1933f1f9.js 39.02kb +0.00b 4.42s +695.00ms ⚠️
dist/component.f31cba5f.js 30.87kb +0.00b 1.94s -1.67s 🚀
dist/esm.621f7fcd.js 27.71kb +0.00b 37.40s +27.52s ⚠️
dist/component.c8d4c7aa.js 22.40kb +0.00b 4.42s +697.00ms ⚠️
dist/js.45b69177.js 16.55kb +0.00b 4.42s +695.00ms ⚠️
dist/ui.dc2b3f59.js 14.11kb +0.00b 10.98s -1.06m 🚀
dist/workerHasher.1a860446.js 11.90kb +0.00b 1.24m +1.09m ⚠️
dist/component.aaadd915.js 6.27kb +0.00b 5.21s -761.00ms 🚀
dist/card.3fd8a427.js 5.71kb +0.00b 13.44s +1.59s ⚠️
dist/media-viewer.b91b6ee1.js 4.01kb +0.00b 13.50s +1.64s ⚠️
dist/EmojiPickerComponent.53446e07.js 3.68kb +0.00b 10.95s +1.60s ⚠️
dist/png-chunks-extract.417e691a.js 3.55kb +0.00b 4.42s +695.00ms ⚠️
dist/dropzone.dcab7b1b.js 3.44kb +0.00b 24.68s +3.46s ⚠️
dist/Modal.d8614cea.js 3.15kb +0.00b 2.46s +1.39s ⚠️
dist/16.4fc0f868.js 2.49kb +0.00b 1.94s +869.00ms ⚠️
dist/ResourcedEmojiComponent.87af091a.js 2.15kb +0.00b 10.95s -25.62s 🚀
dist/card.bfb3f5f8.js 2.15kb +0.00b 22.51s +1.29s ⚠️
dist/date.7c1c4784.js 1.96kb +0.00b 5.01s +665.00ms ⚠️
dist/images.c026df2d.js 1.90kb +0.00b 5.01s -1.20s 🚀
dist/feedback.89b37c2d.js 1.86kb +0.00b 9.72s +4.94s ⚠️
dist/16.5678cfe5.js 1.86kb +0.00b 4.79s +735.00ms ⚠️
dist/16.fb8534c0.js 1.79kb +0.00b 5.47s +1.58s ⚠️
dist/workerHasher.209d7f04.js 1.75kb +0.00b 1.25m +1.08m ⚠️
dist/list-number.f10dae12.js 1.68kb +0.00b 5.11s +626.00ms ⚠️
dist/status.e89e15e9.js 1.68kb +0.00b 5.33s +646.00ms ⚠️
dist/code.17026d12.js 1.61kb +0.00b 4.92s +570.00ms ⚠️
dist/link.5063f523.js 1.53kb +0.00b 5.11s +625.00ms ⚠️
dist/heading6.95167c98.js 1.53kb +0.00b 9.72s +4.93s ⚠️
dist/heading3.59235a88.js 1.51kb +0.00b 8.13s +3.35s ⚠️
dist/16.85df7a6e.js 1.51kb +0.00b 4.57s +676.00ms ⚠️
dist/16.34fd12e7.js 1.46kb +0.00b 5.21s -833.00ms 🚀
dist/16.ff3f3afd.js 1.46kb +0.00b 4.57s +677.00ms ⚠️
dist/emoji.0ed2a831.js 1.45kb +0.00b 5.01s +667.00ms ⚠️
dist/16.55d1de3e.js 1.45kb +0.00b 1.94s +869.00ms ⚠️
dist/16.9689f28f.js 1.44kb +0.00b 4.92s +699.00ms ⚠️
dist/16.ac531af9.js 1.40kb +0.00b 4.92s +699.00ms ⚠️
dist/heading5.83973595.js 1.40kb +0.00b 8.13s +3.35s ⚠️
dist/16.c1346bbe.js 1.36kb +0.00b 4.79s +736.00ms ⚠️
dist/16.341ad0e1.js 1.34kb +0.00b 1.94s +869.00ms ⚠️
dist/heading2.27558830.js 1.33kb +0.00b 5.34s +647.00ms ⚠️
dist/16.be69595e.js 1.32kb +0.00b 4.79s +721.00ms ⚠️
dist/16.0ef0d0ed.js 1.31kb +0.00b 4.92s +700.00ms ⚠️
dist/media-card-analytics-error-boundary.9f8e76f5.js 1.30kb +0.00b 13.44s +1.59s ⚠️
dist/mention.f75f6880.js 1.29kb +0.00b 5.11s +627.00ms ⚠️
dist/heading4.00adf9a0.js 1.29kb +0.00b 8.13s +3.35s ⚠️
dist/Modal.80b784cd.js 1.28kb +0.00b 4.41s +693.00ms ⚠️
dist/16.8015c4e6.js 1.27kb +0.00b 1.94s +869.00ms ⚠️
dist/layout.8d45cad7.js 1.27kb +0.00b 5.11s +625.00ms ⚠️
dist/16.1e3b9f3d.js 1.27kb +0.00b 4.79s +735.00ms ⚠️
dist/16.30604232.js 1.26kb +0.00b 4.57s +675.00ms ⚠️
dist/16.a86c2eba.js 1.26kb +0.00b 4.57s +678.00ms ⚠️
dist/16.8e47d46b.js 1.26kb +0.00b 4.57s +515.00ms ⚠️
dist/16.867d534c.js 1.26kb +0.00b 5.21s -761.00ms 🚀
dist/divider.6089d072.js 1.25kb +0.00b 5.01s +666.00ms ⚠️
dist/quote.f0f2e340.js 1.25kb +0.00b 5.33s +645.00ms ⚠️
dist/component.0a97dc28.js 1.23kb +0.00b 2.46s -1.01s 🚀
dist/action.946aff12.js 1.23kb +0.00b 4.92s +698.00ms ⚠️
dist/16.a43add3b.js 1.23kb +0.00b 2.46s -1.01s 🚀
dist/decision.0a6d7582.js 1.21kb +0.00b 5.01s +665.00ms ⚠️
dist/panel-warning.13de0cff.js 1.21kb +0.00b 5.20s +594.00ms ⚠️
dist/16.2e0d51ab.js 1.18kb +0.00b 4.79s +578.00ms ⚠️
dist/list.36f0ec8a.js 1.18kb +0.00b 5.11s +626.00ms ⚠️
dist/heading1.fcd21443.js 1.18kb +0.00b 5.34s +647.00ms ⚠️
dist/panel-error.36583145.js 1.11kb +0.00b 5.19s +593.00ms ⚠️
dist/panel.529e9757.js 1.10kb +0.00b 5.20s +594.00ms ⚠️
dist/table.402ee877.js 1.09kb +0.00b 5.34s +647.00ms ⚠️
dist/panel-success.83756958.js 1.05kb +0.00b 5.20s +594.00ms ⚠️
dist/media-card-analytics-error-boundary.5602bdf2.js 1.05kb +0.00b 10.99s -1.06m 🚀
dist/panel-note.5ebd6d66.js 1.05kb +0.00b 5.19s +592.00ms ⚠️
dist/media-viewer-analytics-error-boundary.bc280d38.js 995.00b +0.00b 22.51s +1.29s ⚠️
dist/simpleHasher.03c3cf37.js 755.00b +0.00b 1.24m +1.09m ⚠️
dist/index.html 119.00b +0.00b 3.31s -1.00s 🚀

Cached Bundles

Bundle Size Difference Time Difference
dist/index.9413b52a.js 2.31mb +0.00b 594.00ms +71.00ms ⚠️
dist/editorView.2c405cb8.js 587.39kb +0.00b 714.00ms +41.00ms ⚠️
dist/popup.5317d2cb.js 169.21kb +0.00b 688.00ms +38.00ms ⚠️
dist/EmojiPickerComponent.f8cd332f.js 139.08kb +0.00b 686.00ms +59.00ms ⚠️
dist/Toolbar.a1a51fa7.js 99.13kb +0.00b 719.00ms +46.00ms ⚠️
dist/card.a5c85477.js 53.89kb +0.00b 660.00ms +33.00ms
dist/card.00dc27ce.js 51.65kb +0.00b 714.00ms +58.00ms ⚠️
dist/Modal.8e9ccd89.js 38.79kb +0.00b 604.00ms +73.00ms ⚠️
dist/component.f31cba5f.js 30.87kb +0.00b 594.00ms +66.00ms ⚠️
dist/esm.621f7fcd.js 27.71kb +0.00b 687.00ms +60.00ms ⚠️
dist/component.c8d4c7aa.js 22.40kb +0.00b 545.00ms +50.00ms ⚠️
dist/DatePicker.5bd26c52.js 21.03kb +0.00b 686.00ms +86.00ms ⚠️
dist/smartMediaEditor.c1fb8779.js 16.69kb +0.00b 714.00ms +41.00ms ⚠️
dist/js.45b69177.js 16.55kb +0.00b 545.00ms +50.00ms ⚠️
dist/dropzone.59e7fd50.js 15.96kb +0.00b 714.00ms +58.00ms ⚠️
dist/ui.dc2b3f59.js 14.11kb +0.00b 660.00ms +33.00ms
dist/workerHasher.1a860446.js 11.90kb +0.00b 687.00ms +60.00ms ⚠️
dist/component.aaadd915.js 6.27kb +0.00b 545.00ms +50.00ms ⚠️
dist/media-viewer.b91b6ee1.js 4.01kb +0.00b 688.00ms +47.00ms ⚠️
dist/EmojiPickerComponent.8447f1ef.js 3.56kb +0.00b 660.00ms +33.00ms
dist/png-chunks-extract.417e691a.js 3.55kb +0.00b 545.00ms +50.00ms ⚠️
dist/index.cfd8f78a.css 3.46kb +0.00b 723.00ms +55.00ms ⚠️
dist/dropzone.dcab7b1b.js 3.44kb +0.00b 689.00ms +39.00ms ⚠️
dist/Modal.d8614cea.js 3.15kb +0.00b 530.00ms +71.00ms ⚠️
dist/clipboard.a5615679.js 2.97kb +0.00b 714.00ms +58.00ms ⚠️
dist/16.4fc0f868.js 2.49kb +0.00b 530.00ms +72.00ms ⚠️
dist/ResourcedEmojiComponent.87af091a.js 2.15kb +0.00b 660.00ms +33.00ms
dist/card.bfb3f5f8.js 2.15kb +0.00b 688.00ms +38.00ms ⚠️
dist/date.7c1c4784.js 1.96kb +0.00b 617.00ms +52.00ms ⚠️
dist/images.c026df2d.js 1.90kb +0.00b 605.00ms +40.00ms ⚠️
dist/feedback.89b37c2d.js 1.86kb +0.00b 649.00ms +49.00ms ⚠️
dist/browser.dc14c9f2.js 1.82kb +0.00b 714.00ms +58.00ms ⚠️
dist/16.fb8534c0.js 1.79kb +0.00b 560.00ms +45.00ms ⚠️
dist/workerHasher.209d7f04.js 1.75kb +0.00b 687.00ms +60.00ms ⚠️
dist/workerHasher.b62cf8f4.js 1.75kb +0.00b 714.00ms +41.00ms ⚠️
dist/list-number.f10dae12.js 1.68kb +0.00b 617.00ms +38.00ms ⚠️
dist/status.e89e15e9.js 1.68kb +0.00b 626.00ms +46.00ms ⚠️
dist/code.17026d12.js 1.61kb +0.00b 617.00ms +52.00ms ⚠️
dist/link.5063f523.js 1.53kb +0.00b 630.00ms +63.00ms ⚠️
dist/heading6.95167c98.js 1.53kb +0.00b 649.00ms +49.00ms ⚠️
dist/heading3.59235a88.js 1.51kb +0.00b 648.00ms +43.00ms ⚠️
dist/16.34fd12e7.js 1.46kb +0.00b 560.00ms +45.00ms ⚠️
dist/16.ff3f3afd.js 1.46kb +0.00b 560.00ms +45.00ms ⚠️
dist/emoji.0ed2a831.js 1.45kb +0.00b 605.00ms +40.00ms ⚠️
dist/16.55d1de3e.js 1.45kb +0.00b 530.00ms +71.00ms ⚠️
dist/16.9689f28f.js 1.44kb +0.00b 596.00ms +45.00ms ⚠️
dist/16.ac531af9.js 1.40kb +0.00b 596.00ms +47.00ms ⚠️
dist/heading5.83973595.js 1.40kb +0.00b 649.00ms +50.00ms ⚠️
dist/expand.a88194c2.js 1.38kb +0.00b 649.00ms +49.00ms ⚠️
dist/16.c1346bbe.js 1.36kb +0.00b 596.00ms +52.00ms ⚠️
dist/16.341ad0e1.js 1.34kb +0.00b 530.00ms +71.00ms ⚠️
dist/16.be69595e.js 1.32kb +0.00b 596.00ms +52.00ms ⚠️
dist/16.0ef0d0ed.js 1.31kb +0.00b 617.00ms +52.00ms ⚠️
dist/mention.f75f6880.js 1.29kb +0.00b 617.00ms +38.00ms ⚠️
dist/heading4.00adf9a0.js 1.29kb +0.00b 649.00ms +44.00ms ⚠️
dist/Modal.80b784cd.js 1.28kb +0.00b 545.00ms +50.00ms ⚠️
dist/16.8015c4e6.js 1.27kb +0.00b 530.00ms +71.00ms ⚠️
dist/layout.8d45cad7.js 1.27kb +0.00b 607.00ms +42.00ms ⚠️
dist/16.30604232.js 1.26kb +0.00b 560.00ms +45.00ms ⚠️
dist/16.867d534c.js 1.26kb +0.00b 560.00ms +45.00ms ⚠️
dist/divider.6089d072.js 1.25kb +0.00b 605.00ms +40.00ms ⚠️
dist/quote.f0f2e340.js 1.25kb +0.00b 649.00ms +68.00ms ⚠️
dist/component.0a97dc28.js 1.23kb +0.00b 594.00ms +70.00ms ⚠️
dist/action.946aff12.js 1.23kb +0.00b 617.00ms +52.00ms ⚠️
dist/16.a43add3b.js 1.23kb +0.00b 594.00ms +70.00ms ⚠️
dist/decision.0a6d7582.js 1.21kb +0.00b 605.00ms +40.00ms ⚠️
dist/panel-warning.13de0cff.js 1.21kb +0.00b 649.00ms +68.00ms ⚠️
dist/16.2e0d51ab.js 1.18kb +0.00b 596.00ms +48.00ms ⚠️
dist/list.36f0ec8a.js 1.18kb +0.00b 617.00ms +38.00ms ⚠️
dist/panel-error.36583145.js 1.11kb +0.00b 617.00ms +37.00ms ⚠️
dist/panel.529e9757.js 1.10kb +0.00b 649.00ms +68.00ms ⚠️
dist/media-picker-analytics-error-boundary.807445d0.js 1.05kb +0.00b 714.00ms +64.00ms ⚠️
dist/panel-success.83756958.js 1.05kb +0.00b 649.00ms +68.00ms ⚠️
dist/media-card-analytics-error-boundary.5602bdf2.js 1.05kb +0.00b 672.00ms +45.00ms ⚠️
dist/media-card-analytics-error-boundary.fa8679e7.js 1.05kb +0.00b 706.00ms +50.00ms ⚠️
dist/panel-note.5ebd6d66.js 1.05kb +0.00b 622.00ms +41.00ms ⚠️
dist/media-viewer-analytics-error-boundary.bc280d38.js 995.00b +0.00b 688.00ms +38.00ms ⚠️
dist/simpleHasher.03c3cf37.js 755.00b +0.00b 660.00ms +33.00ms
dist/simpleHasher.566203bd.js 755.00b +0.00b 714.00ms +41.00ms ⚠️
dist/index.html 119.00b +0.00b 498.00ms +53.00ms ⚠️

Three.js ✅

Timings

Description Time Difference
Cold 16.77s -107.00ms
Cached 702.00ms -63.00ms 🚀

Cold Bundles

No bundle changes detected.

Cached Bundles

Bundle Size Difference Time Difference
dist/Three.js 580.68kb +0.00b 104.00ms -29.00ms 🚀

Click here to view a detailed benchmark overview.

@mischnic
Copy link
Member

You made casts to any and used invariant(node.type == in many places.
Flow understands if(isIdentifier(..)) { } but not if(t.isIdentifier(..)) { }, so replace every occurence t.is* with is* and import them directly import {isIdentifer} from '@babel/types'.

And NodePath<any> should be NodePath<Node> (import that type from @babel/types).

@devongovett
Copy link
Member Author

Flow understands if(isIdentifier(..)) { } but not if(t.isIdentifier(..)) { }

Oh, what? Why is that?

@mischnic
Copy link
Member

For no particular reason, it's just not implemented: facebook/flow#7735

@mischnic
Copy link
Member

@devongovett Do you still want to merge this? Should I implement the flow changes I suggested?

@devongovett
Copy link
Member Author

Yeah I just haven’t had time yet, sorry!

@mischnic
Copy link
Member

In case we still want to do this:

  1. The babel transformer replaces this in the module scope with undefined (if we don't do this in the transformer, the prelude wrapper could do .call(module.isES6 ? undefined : module.exports))
  2. Multiple reexports with (non)conflicting symbols should be handled gracefully: Transpiling (non-conflicting) namespace reexports emits invalid code babel/babel#11710

# Conflicts:
#	packages/core/integration-tests/test/sourcemaps.js
#	packages/transformers/js/package.json
#	yarn.lock
@devongovett
Copy link
Member Author

Finally had time to update this again. Made it even faster by completely removing @babel/traverse and using the traversal function I developed for the hoist/link perf PRs.

@babel/plugin-transform-modules-commonjs: 8.76s
custom transform using babel-traverse (existing PR): 6.92s (~21% faster)
custom transform without babel-traverse: 4.25s (~51% faster)

@mischnic
Copy link
Member

mischnic commented Dec 8, 2020

We should also make sure that duplicate reexports works correctly (which we fixed previously in Babel's transformer): #4399 (comment)

(So the snippet in that comment should work)

@mischnic
Copy link
Member

Should we remove the parser plugin as well then? At the moment it parses only if the babel transformer doesn't run and then you get an unhelpful assertion error from the babel cjs-esm plugin

#5452 (comment)

@devongovett
Copy link
Member Author

Ah. Well maybe it should be the other way then? Disable the parser plugin by default, but do support it in the transformer. That way people can enable it by adding the syntax plugin to their babel config. If we didn't support transforming it, then they'd also have to use babel's modules transform but that might break some scope hoisting stuff...

@devongovett
Copy link
Member Author

Ok I actually removed all parser plugins from both babel transformer and js transformer. Both dynamic import and export namespace declarations are supported by default now. Export namespace is stage 4 and supported natively in chrome, so I've added support for that in the modules transformer. I added default export support as well, but it must be enabled by adding @babel/plugin-syntax-export-default-from to turn on the parser plugin.

Copy link
Member

@mischnic mischnic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe @wbinnssmith could give this a spin in case we forgot some obvious case?

@mischnic mischnic mentioned this pull request Dec 15, 2020
3 tasks
Copy link
Contributor

@wbinnssmith wbinnssmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this on a codebase that used the following:

export { default } from './foo';

and

export { default as foo } from './foo';

and both are left behind as-is 😞

Copy link
Contributor

@wbinnssmith wbinnssmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@devongovett devongovett merged commit 8fdb958 into v2 Dec 18, 2020
@devongovett devongovett deleted the custom-esm branch December 18, 2020 22:37
AGawrys added a commit that referenced this pull request Jan 7, 2021
Performance improvements (#5461)

Custom faster ESM to CommonJS transformer (#4366)
mischnic added a commit that referenced this pull request Feb 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants