[RFC-0002] Farm's Runtime Module System

[roland-reed]

Farm's Runtime Module System

Abstract

As discussed in #3, Farm will design its own runtime module system to simulate ESM and CommonJS in web browsers, this RFC describes how this module system works.

Support for other module systems, like UMD, SystemJS or AMD, shall be provided by external plugins, Farm itself will not provide now.

In compile time, Farm transforms ESM or CommonJS modules to Farm's standard module, especially, ESM's export/import statements will be replaced as they are defined in ECMA specification, leave it as is will lead to runtime errors.

In runtime, Farm manages all modules with runtime code, in general, a farm's module has three stages in its whole lifetime, which are register, load and run.

Architecture

Farm's module system is composed with two parts:

• Compile time transformation
• Runtime management

Compile time transformation

All modules will be transformed to a CommonJS-like module style:

• Module import: require('REQUEST')
• Module export: exports.foo = 1 / module.exports = {} / module.exports = function () {}

ESM modules will be transformed to CommonJS-like module style, import and export statements defined in ECMAScript specification will be mapped to CommonJS-like module syntax:

Note: require here will be replaced with Farm's actual require implementation of runtime module management.

• ImportDeclaration:
  • import ImportClause FromClause
    • ImportClause:
      • ImportedDefaultBinding:
        • import foo from 'IDENTIFIER' -> const { foo } = require('IDENTIFIER')
      • NameSpaceImport:
        • import * as ns from 'IDENTIFIER' -> const ns = require('IDENTIFIER')
      • NamedImports:
        • import { foo, bar } from 'IDENTIFIER' -> const { foo, bar } = require('IDENTIFIER')
      • ImportedDefaultBinding, NameSpaceImport:
        • import foo, * as ns from 'IDENTIFIER' -> const ns = require('IDENTIFIER'); const foo = ns.default
      • ImportedDefaultBinding, NamedImports:
        • import foo, { foo, bar, baz as z } from 'IDENTIFIER' -> const { default: foo, foo, bar, baz: z } = require('IDENTIFIER')
  • import ModuleSpecifier:
    • import 'IDENTIFIER' -> require('IDENTIFIER')
• ExportDeclaration:
  • export ExportFromClause FromClause
    • export * from 'IDENTIFIER';
      • for every export identifier ex from 'IDENTIFIER', module.exports[ex] = require('IDENTIFIER')[ex]
    • export * as ns from 'IDENTIFIER'; - > exports.ns = require('IDENTIFIER')
    • export { foo, bar, baz as z} from 'IDENTIFIER'; -> exports.foo = require('IDENTIFIER').foo; exports.bar = require('IDENTIFIER').bar; exports.z = require('IDENTIFIER').baz
  • export NamedExports:
    • export { foo, bar, baz as z }; -> exports.foo = foo; exports.bar = bar; exports.z = baz
  • export VariableStatement:
    • export var baz = 1 -> export.baz = 1
  • export Declaration:
    • HoistableDeclaration:
      • export function foo() {} -> exports.foo = function foo() {}
      • export function foo* () {} -> exports.foo= function foo* () {}
      • export async function foo() {} -> exports.foo= async function foo() {}
      • export async function foo* () {} -> exports.foo= async function foo* () {}
    • ClassDeclaration:
      • export class foo {} -> exports.foo= class foo {}
    • LexicalDeclaration:
      • export let foo = 1 -> exports.foo = 1
      • export let foo, bar, baz = 1 -> exports.foo = undefined; exports.bar = undefined; exports.baz = 1
      • export const bar = 1 -> exports.bar = 1
      • export const foo = 1, bar = 1 -> exports.foo = 1; exports.bar = 1
  • export default HoistableDeclaration:
    • export default function foo() {} -> exports.default= function foo() {}
    • export default function foo* () {} -> exports.default= function foo* () {}
    • export default async function foo() {} -> exports.default= async function foo() {}
    • export default async function foo* () {} -> exports.default= async function foo* () {}
  • export default ClassDeclaration:
    • export default class foo {} -> exports.default= class foo {}
  • export default [lookahead ∉ { function, async [no [LineTerminator](https://tc39.es/ecma262/#prod-LineTerminator) here] function, class }] AssignmentExpression:
    • ConditionalExpression:
      • export default foo ? bar : baz -> exports.default = foo ? bar : baz
    • ArrowFunction:
      • export default () => 1 -> exports.default = () => 1
    • AsyncArrowFunction:
      • export default async () => 1 -> exports.default = async () => 1
    • LeftHandSideExpression = AssignmentExpression:
      • export default a = 1 (where a is a variable declared with var or let) -> exports.default = a = 1
    • LeftHandSideExpression = AssignmentOperator AssignmentExpression:
      • export default a += 1 (where a is a variable declared with var or let, AssignmentExpresssion: one of *= /= %= += -= <<= >>= >>>= &= ^= |= **=) -> exports.default = a += 1
    • LeftHandSideExpression &&= AssignmentExpression:
      • export default a &&= 1 (where a is a variable declared with var or let) -> exports.default = a &&= 1
    • LeftHandSideExpression ||=AssignmentExpression:
      • export default a ||= 1 (where a is a variable declared with var or let) -> exports.default = a ||= 1
    • LeftHandSideExpression ??=AssignmentExpression:
      • export default a ??= 1 (where a is a variable declared with var or let) -> exports.default = a ??= 1

Examples

Runtime management

Todo

Examples

[wre232114]

We just need to transform esm to commonjs I think, and leave commonjs unchanged, we can provide a commonjs compatible moudle system, as swc itself supports transforming esm to commonjs, so we can save a lot of work.

And I think, the final generated file's module system would be esm by default(with web and commonjs format supported for compatibility), and it wraps our commonjs like module system which contains many modules. So we can use native esm support of browser and nodejs to fetch/load our final generated file but without losing original module granularity.

[roland-reed]

Yes, but there are still some ESM export or import statements that CommonJS cannot support, for example, top-level await export (export default await 1), these situations should be avoided during compilation (I'm not sure whether SWC has this feature or not).

ESM or CommonJS wrapper will be described in this RFC.

[wre232114]

I have an idea, dynamic import will be transformed to import('final generated file').then(file => file.load(id)), for example:

function a() {
    import('./b').then(module => console.log(module))
}

will be transformed to

function a() {
    import('vendors/xxxx.mjs').then(file => file.load('./b')).then(module => console.log(module))
}

This way we do not need to design how to load files in browser or node.js, they both native support esm, top level await is also not a problem any more. I think we may drop commonjs support for nodejs and only support esm for SSR, only provide web target for legacy browsers.

In browser, all the inital modules will be registered in initial html script tags, dynamic modules will be loaded by import('./xxx').
In node, insert all resource dependencies via import 'xxxx.mjs' to register all dependencies modules.

And the generated file looks like:

// if the target is node, will load static resource dependency as below
import 'xxxxxx.mjs'; // only add this in node.js

globalThis.registerModules({
    './b': function(module, exports, require) {
         exports.b = 'b';
         // dynamic load module c
         const C = import('c.hash.mjs').then(file.load('./c'));
         const d = require('./d');
    },
   './d': function(module, exports, require) { /* ... */ }
});

export function load(moduleId) {
    return globalThis.requireModule(moduleId);
}

[roland-reed]

In browser, all the initial modules will be registered in initial html script tags, dynamic modules will be loaded by import('./xxx').

For browsers don't support import(), we still need a global namespace to register and load modules.

But how to transform top-level await with a module exported by another one in legacy browsers? For example:

export default await import('another-module')

The module above should export a sync default export, not a promise. I cannot find a way to load this module correctly in legacy browsers.

[wre232114]

For legacy browsers, we transform the modules as follow:

export function a() {
    import('./b').then(module => console.log(module))
}

export default await import('d')

will be transformed to

globalThis.registerModules({
    './a': async function(module, exports, require, __farm_dynamic_import__) {
        exports.a = function a() {
            __farm_dynamic_import__ ('vendors/xxxx.mjs').then(file => file.load('./b')).then(module => console.log(module))
        }

       exports.default =  await __farm_dynamic_import__('...') ...;
    },
});

__farm_dynamic_import__ is our polyfilled dynamic import, we may use jsonp to load modules. Farm will also provide internal async polyfill.

for top level await, the module will be transformed to a async function, and our module system will recognize async module function and load and execute it asynchronizely

[roland-reed]

But for the importer of module 'a', the module initializer is still an async function (returning a promise), the importer still have to use Promise.then() instead of using it directly.

// source code of b
export default 'b';

// source code of a
export default await import('b');

// transformed module a
globalThis.registerModules({
    './a': async function(module, exports, require, __farm_dynamic_import__) { // but this function returns a promise, not a string
        exports.a = function a() {
            __farm_dynamic_import__ ('vendors/xxxx.mjs').then(file => file.load('./b')).then(module => console.log(module))
        }

       exports.default =  await __farm_dynamic_import__('...') ...; // this is a string value
    },
});

// importer of module a

// source code
import a from 'a';

console.log(a, typeof a); // this should be 'b' and 'string'

// transformed code
const a = moduleMap['a'](); // this is a promise

console.log(a, typeof a); // this should be Promise and object

a.then(v => console.log(v, typeof v)); // this is 'b' and 'string'

So, there is no way to emulate top-level await in environment that doesn't support top-level await natively, we have to throw error when user use it for legacy browsers.

[wre232114]

// importer of module a

// source code
import a from 'a';

console.log(a, typeof a); // this should be 'b' and 'string'

// transformed code
const a = moduleMap['a'](); // this is a promise

console.log(a, typeof a); // this should be Promise and object

a.then(v => console.log(v, typeof v)); // this is 'b' and 'string'

So, there is no way to emulate top-level await in environment that doesn't support top-level await natively, we have to throw error when user use it for legacy browsers.

I think the transformed code of the importer of should look like:

// transformed importer module a
globalThis.registerModules({
    './importer-of-a': async function(module, exports, require, __farm_dynamic_import__) { // but this function returns a promise, not a string
        var a = require('./a').a;
        var other = require('./other');
        // above async modules should be loaded concurrently

       // await a before use
       var a = await a;
       exports.default =  a; // a is a string value
    },
});

This works the same as top-level-await definition (see MDN top level await and How does top level await work under the hood).

Webpack supports top level await too, we may investigate how webpack handle it. Anyway, if a module is a async module, then the parent module will be async too until the root main module. We just need to require the main module as normal:

globalThis.requireModule('./main');

[roland-reed]

Closed in favor of farm-fe/rfcs#1