Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] - Module/nomodule for NextJs #7563

Open
janicklas-ralph opened this issue Jun 12, 2019 · 12 comments

Comments

Projects
None yet
10 participants
@janicklas-ralph
Copy link

commented Jun 12, 2019

WIP

I have these changes working on my branch. I am actively testing the changes with different sites and also trying to improve the build size. I will raise a PR soon, but would love to get any feedback about the approach.

Goal

Ship modern JS to modern browsers.

Is your feature request related to a problem? Please describe.

next build compiles all client side JS with a default babel-preset-env options, which transpiles all ECMAScript 2015+ code. While this ensures support for older browsers, it ends up shipping unnecessary Babel helpers and polyfills to modern browsers.

Transpiling JS to target modern browsers will have a significant impact on bundle size and performance.

Describe the solution you'd like

Generate modern JS files as part of the current build, by compiling them with babel-preset-env options:

[
  [
    "@babel/preset-env",
    {
      "targets": {
        "esmodules": true
      }
    }
  ]
]

With this option, Babel transpiles files to target a modern browser set and generates relatively modern code.

We can then conditionally ship modern/legacy bundle with the module/nomodule pattern. (Read this blog for more info). The module/nomodule pattern provides a convenient and reliable mechanism for serving modern JavaScript to modern browsers while still allowing older browsers to fall back to polyfilled ES5.

<script type="module" src="modern-build.js"> - to load the modern files

<script nomodule src="current-build.js"> - fallback for browsers that don’t support the modern features.

Integrating with NextJS

Create a plugin similar to BabelESMPlugin that integrates into Nextjs and outputs two versions of bundles on build, a default bundle and a modern bundle. This plugin spawns an additional child compiler inside webpack and re-runs the entire build with the modern babel config options.

Optional feature

This feature will be enabled behind a flag in next.config.js. The flag would be modern: true to enable and modern: false to disable.

Things to note

  • I built and tested this plugin locally with a couple of NextJs sites (built using NextJS). Compiling two versions of bundles increase the overall build time slightly. I noticed an average increase of 15-20% in build time. But in general, the build time increase varies a lot based on the project's build config and its size/structure.

  • Some stats on the before/after bundle sizes

Site Size Before Size after
Hello World app 72.64 kb 72.61 kb
geo.data.gouv.fr/en 463.03 kb 456.37 kb
Zeit.co 7.74 MB 7.41 MB
dp.la 653.15 kb 639.09 kb
  • While the size wins from modern compilation might seem small from the table, there are a couple of factors that we need to consider

    • Pre-transpiled node_modules - A large part of the above sites are node_module files. I ran my changes only on user files, since most of the published node_module libs are already transpiled.
    • Default polyfills - Having a modern chunk provides an opportunity to do further/modern optimization. For eg: We can remove some of the default polyfills included in the current NextJs build like fetch/regenerator-runtime.
    • Parse time wins - I don't have quantifying numbers for this, but In the longer term, modern code will be parsed/executed faster than the legacy code.
  • Parallel work - Some additional optimizations that we are looking into

    • Removing/move polyfills from the modern build. NextJs by default ships polyfills like fetch and regenerator-runtime, these can be safely removed from the modern build.

    • node_modules source - We are working to have node_modules (atleast the ones NextJs directly depends on for now) to have them publish non-transpiled code as ESmodules. This gives us a lot of flexibility to choose the level of tranpilation needed for our modern bundle. Further/better optimization can be performed on these modern code such as better dedup, and tree-shaking, etc.

    • Investigation into using useBuiltIns: usage for modern build

Overall, having a modern bundle unlocks a lot of new optimization opportunities that can be further explored.

@timneutkens

This comment has been minimized.

Copy link
Member

commented Jun 12, 2019

Related community issue: #3056

@TxHawks

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

With this approach, it would seem like only including the polyfills in the nomodule bundle should be pretty simple to do and could probably show pretty significant gains in bundle size.

Also, it isn't mentioned in the RFC, but would be nice to allow users to override the modern build's target.

@janicklas-ralph

This comment has been minimized.

Copy link
Author

commented Jun 13, 2019

@TxHawks Yes, you are right. Looking into the polyfills bit now. I definitely want to have them removed when I get the modern build out.

Also, it isn't mentioned in the RFC, but would be nice to allow users to override the modern build's target.

That's a very good point! I will make that configurable.

@mquandalle

This comment has been minimized.

Copy link

commented Jun 14, 2019

For reference, a similar issue on Gatsby: gatsbyjs/gatsby#2114

@azizhk

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2019

So I got the build working with BabelESMPlugin.
Required some changes for which I've opened a PR: prateekbh/babel-esm-plugin#21
Had to inline babel config in next.config.js and had to comment this line:

} else {
// Add our default preset if the no "babelrc" found.
options.presets = [...options.presets, presetItem]
}

Changes to next.config.js can be found here: azizhk/next-typescript-mdx-emotion#3
I'll work on NextScript and Page Loader next. Then I need to use node_modules from their original source and not ES5 pre-transpiled npm packages.

@timneutkens

This comment has been minimized.

Copy link
Member

commented Jun 18, 2019

Ralph actually already has a working branch that he’s going to push up soon. He’ll share all the details once that is up, but it uses a different approach 👍🏻

@JoviDeCroock

This comment has been minimized.

Copy link

commented Jul 8, 2019

Hey,

When writing https://github.com/jovidecroock/webpack-module-nomodule-plugin I noticed that module-nomodule building will result in the following:

  • both bundles downloaded on IE11 (only nomodule executed)
  • three downloads in classic Edge

This in my opinion is "acceptable" in terms that you are supporting the older browsers and rewarding the newer ones.
In terms of the node_modules problem there is light on the horizon, currently a movement has started in developit/microbundle#413. This output would be used in the pkg.json, at this moment there is no resolving support for it yet but I've written a bit of a placeholder for it https://github.com/JoviDeCroock/webpack-syntax-resolver-plugin

Just thought to put this out there, glad to see how much effort is being put into this all. Great work on Next9 big congratulations on the launch!

@mrksbnch

This comment has been minimized.

Copy link

commented Jul 8, 2019

I'm not sure if that has already been considered but Nuxt.js has a modern option and uses an additional setting to perform a server-side check whether to include the module or nomodule version. This at least prevents the double or even tripple fetching in certain browsers.

@developit

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2019

Differential Loading in Next.js

TL;DR: we'd like to serve modern JavaScript to ES Modules-supporting browsers but fall back to transpiled JavaScript, using the module/nomodule pattern.
However, the HTML version of this approach suffers from a few browser issues that cause duplicated requests for script resources in a small percentage of browsers. There is also a Safari bug requiring a small workaround to prevent duplicate execution. Let's explore some options for dealing with these issues in Next.js.

Part 1: Add the Safari Workaround

Most of the issues noted above come down to fetching unnecessary resources, or duplicate copies of resources. Safari 10.1, however, actually executes both the modern and legacy scripts - that's not a performance tradeoff, it's a bug. Safari 10.1 usage is almost non-existent at this point, but the workaround is relatively small and unobtrusive.

Ideally, it'd be possible to control whether this workaround is included through Next's configuration. Failing that, I think it's probably safest to include the workaround for now.

Part 2: Consider a JS variant of module/nomodule

As described in my recent post, there are fewer issues to be dealt with when loading module VS class scripts from JavaScript than from
HTML. The trade-off here is that you lose out on the ability for the preload scanner to detect and start fetching resources while the HTML document is being streamed. However, I believe Next might be in a position where this doesn't matter, since it already generates <link rel=preload> tags for every
script resource places them in <head>. The preload scanner will find and fetch these, which means the corresponding <script> tags aren't necessary to get the benefit of early resource discovery.

Here's an approximation of Next's HTML today:

<head>
  <link rel=preload as=script href=runtime.js>
  <link rel=preload as=script href=page1.js>
  <script src=runtime.js></script>
  <script src=page1.js></script>
</head>

As you can see, there's not much value in the script tags from a preloading standpoint. They are just there to trigger script execution. Here's the output approximation after Ralph's PR:

<head>
  <link rel=preload as=script href=runtime.esm.js>
  <link rel=preload as=script href=runtime.js>
  <link rel=preload as=script href=page1.esm.js>
  <link rel=preload as=script href=page1.js>
  <script type=module src=runtime.esm.js></script>
  <script nomodule src=runtime.js></script>
  <script type=module src=page1.esm.js></script>
  <script nomodule src=page1.js></script>
</head>

Firstly, it seems like we could make a case for not preloading those legacy bundles. The vast majority of browsers will be downloading but never executing them. Second, that's a fair bit of duplication! If we switched to a simple JavaScript-based script loader, we could do away with the duplication, but we
don't give up the benefits of the preload scanner because we've still got those <link rel=preload> tags.

Option: Direct Implementation of $loadjs

Taking the ESM example from above, we can factor out the loading logic into a JS function:

<head>
  <!-- ignore preloading for now -->
  <script>
    // here’s our little script loader.
    function $loadjs(src, fallback, s) {
      s = document.createElement('script');
      // if there’s nomodule support, load the modern version:
      if ('noModule' in s) s.type = 'module', s.src = src;
      else s.src = fallback;  // otherwise give it the legacy fallback
      document.head.appendChild(s);
    }

    // now, we just have to output function calls for each script asset:
    $loadjs("runtime.esm.js", "runtime.js");
    $loadjs("page1.esm.js", "page1.js");
  </script>
</head>

This could be extended further, since the URLs of the modern and legacy bundles always follow the same pattern - we can generate the modern bundle URL from the legacy one:

<script>
    function $loadjs(src, s) {
      s = document.createElement('script');
      if ('noModule' in s) s.type = 'module';
      else src = src.replace('.esm', '');
      s.src = src;
      document.head.appendChild(s);
    }
    $loadjs('runtime.esm.js');
    $loadjs(page1.esm.js');
</script>

If it's determined that optimizing for the 95% case is the best way forward, adding <link rel=preload> back completes this solution and brings back the benefits of the preload scanner. It also could make it possible to automatically inject script tags for all the preload meta tags, avoiding duplication altogether:

<head>
  <link rel=preload as=script href=runtime.esm.js crossorigin 🐾>
  <link rel=preload as=script href=page1.esm.js crossorigin 🐾>
<script>
    [].map.call(document.querySelectorAll('[🐾]'),function(l,s){
      s=document.createElement('script');
      s.src = 'noModule' in s ? l.href : l.href.replace('.esm', '');
      l.parentNode.appendChild(s);
    })
</script>

I'm not saying using a paw print as the sigil for this would be advisable, it just seemed cute.

@azizhk

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2019

👍 @developit's solution. Just adding a few points which would also need some thought.

  1. Content Security Policy (compute hashes for inline scripts)
  2. Not all js assets have the same path i.e. webpack runtime, commons & chunks have build hashes but this can be handled.
@Ephem

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2019

This is a complex problem and I like @developit's solution and tradeoff. The tradeoff is that some browsers will download both bundles because they support preload, but don't support type=module. When thinking about this I realized that the intersection here can not be that large, so I did some research.

Browsers that support preload: https://caniuse.com/#search=preload
Browsers that support type=module: https://caniuse.com/#search=type%20module

From what I can tell, only Chrome 50-60 and Opera 37-47 supports preload without also supporting type=module.

I downloaded and crunched some numbers from http://gs.statcounter.com/browser-version-market-share and if I did it correctly, the global market share for those browsers:

Last year: 0.98%
Last month: 0.65%

I might have missed something, but seems like this solution is optimizing for the 99.35%. 😄

Only way I can see of improving that number further is using UA-sniffing serverside where possible (that is, only for uncached pages) to determine what script to preload. To me this seems very complex and brittle and not worth it considering the low impact.

@schmod

This comment has been minimized.

Copy link

commented Jul 11, 2019

I don't think "ESM Support === No Polyfills or transforms" is necessarily a great assumption.

Drawing the line between "modern browsers" and "legacy browsers" is by no means black and white, and I'm not sure there's a single feature that can safely serve as the dividing-line.

Safari 10 is a notable example, as it supports ES Modules, but has a broken async implementation (and thus typically requires the async-to-generator transform).

I noted in another PR that this specific issue can have a measurable impact on bundle sizes when babel-preset-env's esmodules option is used to target modern browsers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.