Pre-RFC: Generate and serve ES2015 assets

[tomdale]

This is a pre-RFC because I don't have the bandwidth to write up a full RFC, so if someone wants to pick this up that would be awesome. 😁

Generate and serve ES2015 assets

Summary

Produce both modern and legacy (ES5) versions of JavaScript assets, and intelligently feature detect which version to load using <script type="module"> and <script nomodule> tags.

Motivation

Ember CLI currently produces a single version of an application's JavaScript assets, transpiled to the "lowest common denominator" browser target. However, transpiled output can be quite a bit larger and slower than when using native syntax. Even though a majority of an application's users may be using evergreen browsers with ES2015 support, we still serve them the bigger, slower ES5 version in order to not break older browsers like Internet Explorer.

The good news is that we can take advantage of two new browser features to detect and load an ES2015-compatible version of JavaScript assets in evergreen browsers, while falling back to the ES5 build in browsers that don't support ES2015.

We rely on the combination of two platform features:

1. <script type="module">, which fetches and evaluates JavaScript files in "module" mode, and is ignored by browsers that don't support modules, i.e., browsers that also don't support other ES2015 features.
2. <script nomodule>, which tells the browser not to fetch the linked resources if it supports ES modules. (Similar in concept to the <noscript> tag, which tells browsers to only show content if JavaScript is disabled.)

In essence, we use the existence of module support (as detected by properly-annotated script tags) as a proxy for overall ES2015 support. In practice, modules were one of the last ES2015 features to be implemented by browsers, so every browser that supports modules also reliably supports the remainder of the ES2015 spec.

Detailed design

A few open questions here.

First, how does this interact with the build targets feature? We could say that Ember CLI produces two versions of JS assets, a "legacy" version and a "modern" version, and allow users to configure the browser targets for each. But this feels bad, because it's hardcoded to the current situation and is not amenable to change in the future.

A better option is probably to extend the build targets API to support multiple targets. Here's a strawman for what this API could look like:

module.exports = [{
  output: {
    dir: 'es2015'
  },
  browsers: [
    'last 1 edge versions',
    'last 1 Chrome versions',
    'last 1 Firefox versions',
    'last 1 Safari versions'
  ]
}, {
  output: {
    dir: 'es5'
  },
  browsers: [
    'ie 11'
  ]
}];

In this example, we now support exporting an array of build targets. We also add an output section to the configuration, with a dir property that specifies which directory this target should be compiled into.

This directory is relative to dist, so the es2015 example above would be compiled into e.g. dist/es2015/app.js. The proposed output syntax is consistent Rollup's configuration format, although as proposed we only support the dir option.

Assuming that the application blueprint defaults to something like the above-listed config/targets.js, we can then update the index.html blueprint to the following:

<script type="module" src="{{rootURL}}assets/es2015/vendor.js"></script>
<script type="module" src="{{rootURL}}assets/es2015/<%= name %>.js"></script>
<script nomodule src="{{rootURL}}assets/es5/vendor.js"></script>
<script nomodule src="{{rootURL}}assets/es5/<%= name %>.js"></script>

The default defer semantics of <script type="module"> may not work for us here as I believe the app will break if vendor.js is not evaluated before app.js, so that is something we would want to investigate.

That said, implementing the above two steps I think is enough to let us start serving smaller, faster assets to the majority of users, and most importantly, without requiring any invasive changes to server configurations.

Next Steps

After writing all of this out, it may be that this should be broken into at least two standalone RFCs:

1. Producing multiple build targets in a single build.
2. Changing the default index.html as described above.

Anyone who is interested in working on this should feel free to pick it up, and ping me on Discord if there are questions or problems with this proposal. 😄

Shoutout to @kristoferbaxter from the Google AMP team for persuading me this was a good idea when I ran into him at 4am in the terminal at Norman Y. Mineta San Jose International Airport. 😂

[ctcpip]

safe to say IE11 is the only mainstream browser that wouldn't support? also, where can we now find the current supported list of browsers (for ember)?

[tomdale]

@ctcpip IIRC our currently supported browser matrix can be found in our .travis.yml. At least, that’s the most honest statement of what we support. 😂

[kristoferbaxter]

Something that might help is using the recently added esmodules target in Babel.

Here's an example: https://github.com/kristoferbaxter/preset-env-modules

Edit: https://github.com/kristoferbaxter/preset-env-modules/blob/master/.babelrc.js#L4 is an example.

[chadhietala]

How does the registry work for the DI container?

[tomdale]

@chadhietala I don’t understand the question.

[chadhietala]

This should clearly outline that is just a proxy for ES2015 and we would continue to use the AMD loader.

[NullVoxPopuli]

when are we dropping support for IE11?

[ctcpip]

@NullVoxPopuli it is supported by MS until 2025. I reckon support could be dropped ahead of that, but market share is still high for IE11. For comparison, Angular still supports IE9 (!)
@tomdale I can't find that browser matrix you referenced. We really need to be explicit about browser support again. (IIRC, this used to be discoverable, and easily so.)

[Turbo87]

I've played around with a similar feature in the vue-cli and it works quite well. I would love to get this into ember-cli but I'm a little worried about the impact it would have on build performance right now.

[tomdale]

@Turbo87 Correct me if I'm wrong, but wouldn't producing two builds be relatively easy to parallelize? In general, I'm concerned that Broccoli is still missing primitives that allow it to detect and parallelize isolated sub-graphs that don't share dependencies—I think that, plus better pruning of transforms "downstream" from pure transforms that don't detect changes, would make it easier for us to experiment with things like this.

Do we have a plan for getting build performance to a place where we could feel comfortable introducing something like this?

[Turbo87]

Correct me if I'm wrong, but wouldn't producing two builds be relatively easy to parallelize?

AFAIK we already try to parallelize some of the work (e.g. Babel transpilation). If all of your CPUs are already busy with Babel, then building a second asset tree in parallel will likely not be as fast as just building a single asset tree.

Do we have a plan for getting build performance to a place where we could feel comfortable introducing something like this?

I don't know the exact details, but I think the packager work was moving in a direction where most files were kept as ESlatest as long as possible and the transpilation to lower browser targets happens relatively late in the pipeline. If the two pipelines ("modern" and "legacy" builds) can share most of the work and only the transpilation step at the end is duplicated this might be fine. But as far as I'm aware that is not quite the case right now as the current pipeline (pre-packager) is quite convoluted...

[simonihmig]

Talked about the dual build performance issue briefly with @Turbo87 at Emberfest yesterday, will try to sum up my thoughts here as well:

• I believe you very rarely would want to have the legacy build in the ember serve scenario, where build performance is most critical, as developers will use browsers that will never use the "legacy" bundle. Only in the specific case, where someone wants to test things in IE there could be an opt in to activate the legacy build (say via a CLI flag).
• For ember test it might make sense to also disable the legacy build by default, as enabling is only useful if your test setup is able to run IE (i.e. something like Browserstack/Saucelabs), which might not be the case for many apps I would guess.
• For ember build -prod you would want to have it included of course. But that's mostly running in CI, so not that big of a performance concern here probably?

[NullVoxPopuli]

So, ember uses babel, and we can configure babel to our heart's content...

Has anyone configured a module build?
I'd like to see how happens to my bundle size :)

[rwjblue]

By default development builds are already mostly evergreen browsers only (check out your config/targets.js). We only build for lowest common denominator in CI and for production.

The feature here would be to enable building two sets of assets for production and using the technique described above (script tag with modules attribute + an array in config/targets.js).

—-

It seems that folks here are forgetting the most straightforward answer here though. The absolute simplest way to do this would be dropping IE11 support completely 😺...

[NullVoxPopuli]

I just mention it because major browsers support async/await, but the transpiled code is (or was, need to check) runtime generator stuff.

[rwjblue]

Ya, but thats just based on what your config/targets.js has. If you only have evergreen browsers listed there already (aka don't support IE11), you will be shipping native async/await (and everything else that modern browsers support out of the box).

[NullVoxPopuli]

well, nevermind then! my bundle does indeed have async/await in it :)
👍

[astronomersiva]

@tomdale, we have been experimenting with this in our apps.

Initially, we based our technique on the Lightning strategy, using two different index.htmls to serve assets.

We tried using the module/nomodule approach but like you pointed out, the defer behaviour of modules makes this impossible in our apps. I am curious as to how Vue makes this work even if vendor.js loads much after app.js finishes loading.

One more issue might be handling the app/config/asset-manifest meta in index.html that is needed for engines to work.

Currently, we are changing our approach to simultaneously generate two different builds with
ember build --environment=production --output-path=legacy & MODERN=true ember build --environment=production --output-path=modern and then merging them into a dist folder. Using posthtml, we extract script tags from modern's index.html and insert them in legacy's index.html. We wrap these tags with some markers and replace them at our server before serving to the user.

<MODERN>
<script src="{{rootURL}}assets/vendor-checksum.js"></script>
<script src="{{rootURL}}assets/<%= name %>-checksum.js"></script>
</MODERN>

<LEGACY>
<script src="{{rootURL}}assets/vendor-checksum.js"></script>
<script src="{{rootURL}}assets/<%= name %>-checksum.js"></script>
</LEGACY>

[Turbo87]

The default defer semantics of <script type="module"> may not work for us here as I believe the app will break if vendor.js is not evaluated before app.js, so that is something we would want to investigate.

@tomdale @astronomersiva I'm not sure I understand this part. defer is still supposed to evaluate the scripts in the order that they appear in the file. async might be what you are talking about?

somewhat related: ember-cli/ember-cli#7632

/cc @NullVoxPopuli

[astronomersiva]

@Turbo87, you are right. There was another script tag(that stays the same in both legacy and modern versions because it just does window.something = { something else }) we had that depended on vendor.js and this was the actual reason behind the errors in our end. So, if we decide to go down the module/nomodule path, we have to ensure that all script tags are converted to contain a defer attribute.

However, since variables in modules do not get automatically declared globally, there are various errors caused in vendor.js like:

• define is not defined
• mainContext is not defined

[NullVoxPopuli]

anything holding this up?
I recently found: https://siva.dev/ember-differential-bundles/
and that doesn't seem like that much effort to add to the default build, yeah?

[acorncom]

@NullVoxPopuli there's an addon that helps with it as well: https://github.com/pzuraq/ember-cli-babel-polyfills

[mansona]

I've been experimenting with using half of this Pre-RFC with Empress projects because they should all support working with JS turned off. That means that the esmodules build would be the only build that would be run.

Although I have been having some issues trying to test this idea because I don't quite know how to get an Ember app running in the <script type="module"> scenario 🤔

Do we know if Ember supports the esmodule build as the only build right now?

[Turbo87]

nope, the only thing that is supported at the moment is the regular AMD build. implementing what Tom suggests should be unlocked by Embroider, but until then there is not much point in sinking time into this 😉

[lifeart]

https://github.com/lifeart/ember-cli-fallback-builds

[Gaurav0]

@ctcpip IIRC our currently supported browser matrix can be found in our .travis.yml. At least, that’s the most honest statement of what we support. 😂

I guess we don't support iOS Safari and Chrome Android.

[locks]

Closing due to inactivity.

[NullVoxPopuli]

Do all browsers support modules now?
I'd still like to choose an ESM target for bundle output