Compile to SystemJS on the fly #79
Comments
|
A fast in-browser SystemJS transform would be a great project. Unfortunately this project is not it, since it is not a parser, and SystemJS transformation requires a full parser. A wasm-based fast compiler is definitely the future though after Babel for SystemJS transforms. The problem is the lack of funding and companies building this work. I have a few side projects here too, but without funding it's a very hard thing to get off the ground given the number of developer hours compilers take. |
|
Something I've been considering the last few days is how to make a module format that only requires Basically it works like this: Suppose we have the following ES module: import { foo } from 'foo';
import bar from 'bar';
import * as baz from 'baz';
function localFunction() {
}
export default class SomeClass {
constructor() {
console.log(foo);
}
}
export function hositedFunction() {
console.log(bar);
}
export const value = 10 + baz.boz;Now to preserve hoisting without a full parse and search, we can simply wrap this in a generator function (this is effectively v8's internal implementation and what inspired this approach): defineModule('qux', function*() {
const [foo$random1, bar$random2, baz$random3] = yield {
// Because we can determine imports with es-module-lexer we can
// easily get this list
imports: ['foo', 'bar', 'baz'],
namespace: Object.freeze({
__proto__: null,
[Symbol.toStringTag]: 'Module',
// We wrap each export with a getter, this preserves hoisting
// because generators support hoisting
get default() { return SomeClass; },
get hoistedFunction() { return hoistedFunction; },
// Notice this will pass along a ReferenceError if called
// before the following body is evaluated as is expected
get value() { return value; }
}),
};
class SomeClass {
constructor() {
console.log(foo);
}
}
function hositedFunction() {
console.log(bar);
}
const value = 10 + baz.boz;
});So this gets us most of the way, however we still need to be able to access imports values, which again seems like it needs a full parse. But again a solution: the Because we know all names that'll be imported thanks to defineModule('qux', function*() {
const imports$random1 = {
__proto__: null,
get foo() { return foo$random1.foo },
get bar() { return bar$random2.default },
get baz() { return baz$random3 },
}
with (imports$random1) {
const [foo$random1, bar$random2, baz$random3] = yield {
// Because we can determine imports with es-module-lexer we can
// easily get this list
imports: ['foo', 'bar', 'baz'],
namespace: Object.freeze({
__proto__: null,
[Symbol.toStringTag]: 'Module',
// We wrap each export with a getter, this preserves hoisting
// because generators support hoisting
get default() { return SomeClass; },
get hoistedFunction() { return hoistedFunction; },
// Notice this will pass along a ReferenceError if called
// before the following body is evaluated as is expected
get value() { return value; }
}),
};
class SomeClass {
constructor() {
console.log(foo);
}
}
function hositedFunction() {
console.log(bar);
}
const value = 10 + baz.boz;
}
});Other than a couple easily fixable problems (e.g. no strict mode), this strategy should work as a module format that requires no full parsing (just what I haven't tried this out yet, so I'll create an implementation this week, but I feel like it's promising. |
|
Another aspect of the format that requires a parser is that all assignments in modules need to update their live bindings and that circular references must hoist function definitions across module boundaries. These are edge cases few people rely on yes, but edge cases that need to be handled none the less (at least in terms of what this project and SystemJS aim to support). You are welcome to fork and build your own paths though - this is open source! And if you have a comprehensive approach that handles all the edge cases and you want to collaborate I'm all ears. |
Yep that's the trick of using generators + getters, basically functions are hoisted inside generators as per usual, so if I do To perform hoisting stage of module evaluation I'll call next on all modules at once, this ensures the namespaces are all available before evaluation of the body is performed. For live bindings, because removing the import statements effectively turns the lookups into global variable lookups, I can use the |
|
How would this wrapper approach work with reexports? Where eg The other concern is the strict mode handling - although maybe you can embed a |
|
Also you want to look into reexport chaining with export * etc. |
|
By the way, I think these ideas are very interesting... please do share your demo, as it would be interesting to do the perf and edge case checks. If it can provide a path forward for module workers I'm all for it. |
... Six months later I now have a working implementation of what I described. I basically impelemented Module Records in the spec almost to the letter, using my generator technique above to actually evaluate the module. It's definitely not production ready yet and isn't bundled or minified or anything, however it's ready enough to try playing around with. I wrote a short README with build steps if you wanna try it out. The library can be found here: https://github.com/Jamesernator/module-shim |
|
Nice to see! If you ever want to collaborate on a version 2 of the SystemJS module format specification incorporating these ideas do let me know. |
|
If you want to put your project to the test, see if you can run the SystemJS test suite execution cases here - https://github.com/systemjs/systemjs/blob/master/test/system-core.js#L278. |
I would be interested, yes. I should note the internal format I'm using is slightly different than what is described above. The new format is fairly simple, the following ES Module is converted like so: import SomeImport from "./mod1.js";
import { SomeImport2 } from "./mod2.js";
const someResource = new URL("./foo.yaml", import.meta.url);
export default class FooBar {
load(module) {
return import(module);
}
}
export function foo() {
return 12;
}becomes // Imported bindings are attached to this object, these were detected by the lexer
// so it's guaranteed these are all known ahead-of-time, this acts as the module scope
// Also attached is a generated name for import() and import.meta
// This allows us to avoid a complicated transform that involves analysing scopes
// for import references
with (arguments[0]) {
// Because the with() statement is sloppy-only we need an inner
// wrapper to set strict mode for the module body
yield* function*() {
"use strict";
// These getters are generated by the exports detected by the lexer
// thanks to the behaviour of generators we can access hoisted bindings
// such as foo even before the statements after this yield are executed
// this allows us to match the hoisting behaviour of ES modules without
// a complex transform
yield {
get default() {
return FooBar;
},
get foo() {
return foo;
},
};
// The body is inserted pretty much as is except with import/export
// statements stripped and import()/import.meta replaced to refer to
// a name on the module scope object
const someResource = new URL("./foo.yaml", importMeta$$0.url);
class FooBar {
load(module) {
return dynamicImport$$0(module);
}
}
function foo() {
return 12;
}
}();
}Adapating to an ahead-of-time format would probably just involve skipping the
I should note that my module-shim doesn't actually implement SystemJS, it just implements I do intend to run the module tests from test262 though to ensure it is a spec-compliant implementation of ES Module Records (except for parse errors, as that's just the limitation of the lexer). |
|
Each of those SystemJS tests I pointed to are just ES module cases. Switching those same tests to run against https://github.com/systemjs/systemjs/tree/master/test/fixtures/es-modules would do the same. Put it this way, if you can cover all the test262 tests, that's the bar. |
|
I've run the library against test262 and have all tests passing except for a couple minor things that depend on full parse (so similar to I've published an initial version to Next up is to add support for Top-Level await and arbitrary string import/export names. |
|
@Jamesernator nice to see this, but it doesn't seem to handle the cases I mentioned. The following test just failed for me: import SourceTextModule from 'module-shim/SourceTextModule';
const m1 = await SourceTextModule.create({
source: `
import { f2, v2 } from 'm2';
export function f1 () {
v2++;
}
export var v1 = 10;
console.log(v1, v2);
f2();
console.log(v1, v2);
`,
resolveModule
});
const m2 = await SourceTextModule.create({
source: `
import { f1, v1 } from 'm1';
f1();
export function f2 () {
v1++;
}
export var v2 = 20;
`,
resolveModule
});
function resolveModule (specifier) {
if (specifier === 'm1') return m1;
if (specifier === 'm2') return m2;
}
await m1.link();
m1.evaluate();I would suggest using a simpler linking API of |
|
That is: function resolve(specifier, parentModule)that would match the |
I'm a bit confused, the behaviour of the test case you gave above is the same as running those modules as ESM in Node. Is there some other behaviour you were expecting? module-shimNode
Currently the API is just a mirror of Abstract Module Records by providing the four methods:
It should be noted |
|
Ok, well that is just incredible work then! The only remaining concern I have is if the with / getter approach has a performance cost over setters. Consider eg: import fn from 'dep';
for (let i = 0; i < 1000; i++)
fn(i);if on every tick of the loop it is running a getter it seems like that will be slower than the setter approach used in the System module format currently. Have you done any checks on these types of cases? |
|
Also if there are reexports in the getter chain does that also add to the slowdown? |
I haven't done any checks on the performance yet, but yes I do expect it to be slower due to the An ahead-of-time format could use setters however. I'll have to try writing an evaluator to check, but I think it would easy enough to support both formats (including mixing) without performance loss if the on-the-fly isn't used.
No, all recursive lookups happen only once during the creation of the namespace object. The getters themselves don't perform any recursive lookup. |
|
Yes I think that a replacement for the SystemJS module format itself would need to use a AOT setter approach still, but that would still bring the format simplification of the generator style to the output. I like that you construct the full namespace upfront as well. In terms of eg supporting workers in this project it really depends on how long it takes for dynamic import support in workers to land across the browser baseline. If we get that then this project can rely on that fine. But if that continues to be delayed then your approach might be better. It's still a little difficult to tell which way things will go here so I'd suggest separate projects for the time being, but I'm all for a merge / supporting your project in future. |
|
Well, if the object that’s passed to the |
My interest for this is primarily for module workers in browsers that already support modules, but it would apply to running ES modules in browsers that don't even support ES modules.
The text was updated successfully, but these errors were encountered: