Support ES module extra for production non-reentrant interop

[MicahZoltu]

// index.html
<script src='vendor/systemjs/system.js'></script>
<script>
System.import('/index.js')
</script>

//index.js
import 'knockout'

SyntaxError: import declarations may only appear at top level of a module

[guybedford]

We can definitely provide a native-module extra that will use a <script type="module"> instead of a normal script for System.register. Then just like we fall back for globals we can fall back to the module value if there is no System.register registration.

[joeldenning]

@MicahZoltu you can use the transform loader to convert ES modules to System.register. However, it is not possible for SystemJS to polyfill ES modules themselves without converting them to a differnt format.

The reason that native modules are not polyfillable is because of the import and export keyword throwing syntax errors in browsers that don't support them. Even in the browsers that do support es modules, there's no way for us to add in the extra functionality such as import maps that those browsers are lacking.

For those reasons, all SystemJS modules must eventually be converted to System.register format in order to be used by SystemJS. The AMD, UMD, and global extras are simply a conversion of those formats to System.register. And the transform loader that I referenced above is a way of turning ES modules (or other formats) into System.register during an in-browser compilation step.

What Guy suggested would allow you to load modules with the native module extra, but then you could not interact with those modules in any of the ways that you normally can with SystemJS, including import maps, System.get(), System.has(), etc.

[MicahZoltu]

I'm able to use es-module-shims to interact with native modules while using import maps with no transformation, so I'm confused why I can't do the same in SystemJS.

For context, the problem I'm actually trying to solve is I have some CommonJS modules that I want to use in a project that is targeting evergreen browsers. I'm currently using es-module-shims to get import maps and I was trying to use SystemJS to deal with the CommonJS modules I had, ideally without having to have a transpilation step at build time. Ultimately, I may end up going the transpilation route, I was just hoping that SystemJS was an alternative solution to my problem.

[joeldenning]

es-module-shims is doing in-browser compilation, similar to when you use the transform loader in SystemJS. It has a full blown JavaScript tokenizer that runs in the browser on your code as a string. So your desire to avoid a compilation step is not really possible — it can either be at build-time or in-browser.

See https://github.com/guybedford/es-module-shims/blob/master/README.md#implementation-details

Regarding your CommonJS modules, Guy’s suggestion of creating a native extra for systemjs would allow your CommonJS modules to import native modules. However, afaik it would not allow for native modules to import the CommonJS modules. This is because there would be two separate module registries — the systemjs registry and the browser’s registry. There is no way of getting something into the browser’s registry without having an ES module.

If the native extra would suit your needs, we could consider adding the extra. But I believe it would have the limitations I described above.

[viT-1]

Workaround:

• include node_module file in tsconfig (e.g vue-class.component.esm.js)
• replace registered module names from straight paths to node_module names after bundling

[guybedford]

For modules without imports, or for explicit edges of the graph where the use case of SystemJS is supporting ES modules alongside other module formats, effectively treating the ES module graph as the primary target, but having some CJS / AMD edges, this would at least support reentrancy from that CJS/AMD back into ESM, by having a module like export default await System.import('amd').

I'm open to the use case if it would be useful, it's a simple extra to add.

[LarsDenBakker]

It would be really useful if you could import UMD modules from es module in that way, it would open up quite a migration path for development using just es modules.

[guybedford]

So the way to do this in browsers that support ES modules is not to try to "add UMD support", but simply to do a module exactly like I provided - export default await System.import('umd-module').

If it is a UMD that does a global definition it is even simpler though:

import './global-script.js';
export default globalName;

[MicahZoltu]

Hmm, I would be fine with CommonJS modules not being able to import ES modules, but I definitely need the ability for ES modules to be able to import CommonJS modules. I'm trying to migrate to evergreen browser targeting, but every so often there is some library that would be very difficult to recreate that only exports a CommonJS module (and its transitive dependencies are all CommonJS), so I cannot import it with ES.

I think ultimately what I need is a tool that will transpile CommonJS into ES modules, though I have a suspicion that that isn't actually possible (at least not without any degree of reliability. I suspect what I'll end up doing is just taking the time to recreate whatever features I need from those modules from scratch and deploy it as an ES module. 😬

[mk-pmb]

SyntaxError: import declarations may only appear at top level of a module

I'm struck with the same error after upgrading from systemjs v0.20.19, where I was able to use the babel plugin. I guess I should add the transform loader from extras then? I'm trying to figure out how to do that, but the readme link "pluggable extras" just goes to the folder. Which isn't a bad thing though: Could we add a dist/extras/README.md that explains how to add them?

[mk-pmb]

With the AMD extra, at least something seems to be loaded: Adding a script tag

<script src="../../../../systemjs/dist/extras/amd.min.js"></script>

changes the error from "define is not defined" to "AMD require not supported". So is a simple script tag the way it's meant to be loaded? Indeed the source does look like it's a monkeypatch to be "just thrown in" after dist/system.min.js, but then why would both of them fail?

[guybedford]

It sounds like the following pattern is being used - define(function (require) { require('./x.js'); }); where the above form of AMD expects a regular expression to detect the require statements. This is what is referred to by "AMD require not supported".

…

On Fri, 1 Nov 2019 at 13:03, M.K. ***@***.***> wrote: With the AMD extra, at least *something* seems to be loaded: Adding a script tag <script src="../../../../systemjs/dist/extras/amd.min.js"></script> changes the error from "define is not defined" to "AMD require not supported". So is a simple script tag the way it's meant to be loaded? Indeed the source does look like it's a monkeypatch to be "just thrown in" after dist/system.min.js, but then why would both of them fail? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2013?email_source=notifications&email_token=AAESFSUNEADSZ4VTBVRBA7LQRROO3A5CNFSM4ISSCKN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC3RGWQ#issuecomment-548868954>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAESFSQOZBFQRNW7VUV7RL3QRROO3ANCNFSM4ISSCKNQ> .

[mk-pmb]

Yes indeed. I'm trying to upgrade my demo at https://github.com/mk-pmb/tryjs/blob/gh-pages/module_systems/systemjs/format-guessing/ which had worked with systemjs v0.20.19.

[guybedford]

Yes, so SystemJS stripped back all features, the initial plan was that users would request features that were absolutely necessary and we could add them back. It turned out that there were few requests so the project has stayed small - eg we still don't implement an all-in-one Babel plugin, CommonJS loading, require in AMD. All those things are possible with just some work. But it will be a hard time doing the conversion without the old features without fully embracing that simplification.

[mk-pmb]

I see. Actually for the libs I use, a full babel would probably be overkill anyway. I'll try and investigate how things are meant to work, then write my own transform plugin.

[guybedford]

Yeah it's worth exploring these things. I'm also currently considering if the hooks in #2058 would be a better model than the "transform" hook. Fetch hook is more general as it has streaming and header control, while needing less code in SystemJS (as in you don't need an extra to get it once that PR merges).

Best thing to keep in mind is staying as close to native specs as possible. Fetch hook is nice because it can always be moved to service worker too.

[privatenumber]

ESM is native and is growing increasingly popular. As already mentioned, even though there isn't a native way to resolve bare specifiers, esm-module-shims offers browser compatibility and import-map support for feature parity.

To add, I'm finding that npm packages with ESM distributions are pre-bundled or using relative paths (eg. lodash-es). There are also services such as Pika CDN that has been helpful in bundling packages to ESM on-demand and automatically resolving bare-specifiers.

If the native extra would suit your needs, we could consider adding the extra. But I believe it would have the limitations I described above.

@joeldenning
It seems it's can be done pretty easily. Is it alright if I open a PR for it?

As @LarsDenBakker mentioned, this could offer a nice preliminary step for helping apps migrate to ESM.

[joeldenning]

Is it alright if I open a PR for it?

Sure, that would be great. I took a look at your codepen and it looks like a good start. A couple notes on it:

1. Dynamic import and <script type="module"> aren't supported in any version of IE / Edge except the new ones that use Chromium, as far as I can tell (1, 2). So this extra would only work in the very newest versions of Edge.
2. Guy Bedford suggested using <script type="module"> as an alternative to import() in Support ES module extra for production non-reentrant interop #2013 (comment), which I think would make the implementation perhaps even simpler. I'm not sure how to get the module value out of that script, though, which would be needed. This stack overflow answer and this github comment seem to indicate that there is no way to access the exports out of <script type="module"> elements. @guybedford are you aware of a way to get the exports out of a script element? Or was your previous comment perhaps indicating using both <script type="module"> and import()?
3. Instead of hooking instantiate, you might be able to hook createScript and getRegister. This has some benefits of code reuse between normal scripts and module scripts, along with making the implementation quite simple. The createScript hook would set the type property to "module", and the getRegister hook would either get the exports out of the <script type="module"> element or call import()

[privatenumber]

@joeldenning

I'll open a PR with the code I have so far as a starting point. We can take the discussion
there. Thanks!

[guybedford]

Due to the reentracy issues, I think simply using the Babel plugin is the best option here. Over time we can focus on reducing the overhead of the Babel plugin, in the same way that ES module shims uses a small lexer maybe we will be able to use a very fast System rewrite. @Jamesernator has done some interesting work here in https://github.com/Jamesernator/module-shim which we may be able to adopt as a replacement to the Babel plugin in future.

[joeldenning]

I don't think in-browser compilation will ever be a viable production strategy, even with further optimizations to babel. For that reason, I don't see the systemjs babel plugin as an equivalent to the ESM extra being discussed here.

The reentrancy issue is valid, though.

[guybedford]

Agreed and it's certainly still possible to create an extra for ES module loading without reentrancy, and I'd still be glad to see a PR for that.

Let me reeopen to track just those use cases specifically then.

[joeldenning]

Vue 3 no longer publishes a UMD build, and it cannot be loaded with SystemJS' global loading. As a result, being able to load it with ESM is desireable. See #2272.

Additionally, there has been recent interest in the single-spa community to better support tools like Vite and Snowpack, which has me exploring systemjs/esm interop. See https://single-spa.js.org/docs/ecosystem-vite

it's certainly still possible to create an extra for ES module loading without reentrancy, and I'd still be glad to see a PR for that.

@guybedford doesn't #2187 do just that? Was the reason that PR was closed due to reentrancy issues? Or was it because of TLA? I'm interested in trying to revive ESM loading.

[mk-pmb]

In case it helps, I'll set a bounty of 15€ on the first ESM loading solution that convinces me. (Yeah weak criterion, but hey.) (also SEPA bank transfer only, target must be legal to pay etc.)

[CoooWeee]

angular 13 also no longer publishes a UMD build.
Anyone any solutions how to load ESMs?

[joeldenning]

it's not about how to load ES modules, but that doing so results in two module registries: the systemjs registry and the browser ES module registry. The two module registries cannot share dependencies, which is the main reason why we haven't implemented this.