diff --git a/doc/design/overview.md b/doc/design/overview.md index 38c5d0b..4b5ad82 100644 --- a/doc/design/overview.md +++ b/doc/design/overview.md @@ -19,7 +19,7 @@ Custom loaders are intended to chain to support various concerns beyond the scop ### Proposals -* [Recursive chaining](./proposal-chaining-recursive.md) +* [Chaining Hooks “Middleware” Design](./proposal-chaining-middleware.md) ## History diff --git a/doc/design/proposal-chaining-middleware.md b/doc/design/proposal-chaining-middleware.md new file mode 100644 index 0000000..41a481c --- /dev/null +++ b/doc/design/proposal-chaining-middleware.md @@ -0,0 +1,265 @@ +# Chaining Hooks “Middleware” Design + +## Chaining `resolve` hooks + +Say you had a chain of three loaders: + +1. `unpkg` resolves a specifier `foo` to an URL `http://unpkg.com/foo`. +2. `http-to-https` rewrites that URL to `https://unpkg.com/foo`. +3. `cache-buster` takes the URL and adds a timestamp to the end, like `https://unpkg.com/foo?ts=1234567890`. + +The hook functions nest: each one must always return a plain object, and the chaining happens as a result of each function calling `next()`, which is a reference to the subsequent loader’s hook. + +A hook that fails to return triggers an exception. A hook that returns without calling `next()`, and without returning `shortCircuit: true`, also triggers an exception. These errors are to help prevent unintentional breaks in the chain. + +Following the pattern of `--require`: + +```console +node \ + --loader unpkg \ + --loader http-to-https \ + --loader cache-buster +``` + +These would be called in the following sequence: `cache-buster` calls `http-to-https`, which calls `unpkg`. Or in JavaScript terms, `cacheBuster(httpToHttps(unpkg(input)))`. + +Resolve hooks would have the following signature: + +```ts +export async function resolve( + specifier: string, // The original specifier + context: { + conditions = string[], // Export conditions of the relevant `package.json` + parentUrl = null, // The module importing this one, or null if + // this is the Node entry point + }, + next: function, // The subsequent `resolve` hook in the chain, + // or Node’s default `resolve` hook after the + // last user-supplied `resolve` hook +): { + format?: string, // A hint to the load hook (it might be ignored) + shortCircuit?: true, // A signal that this hook intends to terminate + // the chain of `resolve` hooks + url: string, // The absolute URL that this input resolves to +} { +``` + +### `cache-buster` loader + +
+`cache-buster.mjs` + +```js +export async function resolve( + specifier, + context, + next, // In this example, `next` is https’ resolve +) { + const result = await next(specifier, context); + + const url = new URL(result.url); + + if (url.protocol !== 'data:')) { // `data:` URLs don’t support query strings + url.searchParams.set('ts', Date.now()); + } + + return { url: url.href }; +} +``` +
+ +### `http-to-https` loader + +
+`http-to-https.mjs` + +```js +export async function resolve( + specifier, + context, + next, // In this example, `next` is unpkg’s resolve +) { + const result = await next(specifier, context); + + const url = new URL(result.url); + + if (url.protocol === 'http:') { + url.protocol = 'https:'; + } + + return { url: url.href }; +} +``` +
+ +### `unpkg` loader + +
+`unpkg.mjs` + +```js +export async function resolve( + specifier, + context, + next, // In this example, `next` is Node’s default `resolve` +) { + if (isBareSpecifier(specifier)) { // Implemented elsewhere + return { url: `http://unpkg.com/${specifier}` }; + } + + return next(specifier, context); +} +``` +
+ +## Chaining `load` hooks + +Say you had a chain of three loaders: + +* `babel` transforms modern JavaScript source into a specified target +* `coffeescript` transforms CoffeeScript source into JavaScript source +* `https` fetches `https:` URLs and returns their contents + +Following the pattern of `--require`: + +```console +node \ + --loader babel \ + --loader coffeescript \ + --loader https +``` + +These would be called in the following sequence: `babel` calls `coffeescript`, which calls `https`. Or in JavaScript terms, `babel(coffeescript(https(input)))`: + +Load hooks would have the following signature: + +```ts +export async function load( + resolvedUrl: string, // The URL returned by the last hook of the + // `resolve` chain + context: { + conditions = string[], // Export conditions of the relevant `package.json` + parentUrl = null, // The module importing this one, or null if + // this is the Node entry point + resolvedFormat?: string, // The format returned by the last hook of the + // `resolve` chain + }, + next: function, // The subsequent `load` hook in the chain, + // or Node’s default `load` hook after the + // last user-supplied `load` hook +): { + format: 'builtin' | 'commonjs' | 'module' | 'json' | 'wasm', // A format + // that Node understands + shortCircuit?: true, // A signal that this hook intends to terminate + // the chain of `load` hooks + source: string | ArrayBuffer | TypedArray, // The source for Node to evaluate +} { +``` + +### `babel` loader + +
+`babel.mjs` + +```js +const babelOutputToFormat = new Map([ + ['cjs', 'commonjs'], + ['esm', 'module'], + // … +]); + +export async function load( + url, + context, + next, // In this example, `next` is coffeescript’s hook +) { + const babelConfig = await getBabelConfig(url); // Implemented elsewhere + + const format = babelOutputToFormat.get(babelConfig.output.format); + + if (format === 'commonjs') { + return { format, source: '' }; // Source is ignored for CommonJS + } + + const { source: transpiledSource } = await next(url, { ...context, format }); + const { code: transformedSource } = Babel.transformSync(transpiledSource.toString(), babelConfig); + + return { format, source: transformedSource }; +} +``` +
+ +### `coffeescript` loader + +
+`coffeescript.mjs` + +```js +// CoffeeScript files end in .coffee, .litcoffee or .coffee.md. +const extensionsRegex = /\.coffee$|\.litcoffee$|\.coffee\.md$/; + +export async function load( + url, + context, + next, // In this example, `next` is https’ hook +) { + if (!extensionsRegex.test(url)) { // Skip this hook for non-CoffeeScript imports + return next(url, context); + } + + const format = await getPackageType(url); // Implemented elsewhere + + if (format === 'commonjs') { + return { format, source: '' }; // Source is ignored for CommonJS + } + + const { source: rawSource } = await next(url, { ...context, format }); + const transformedSource = CoffeeScript.compile(rawSource.toString(), { + bare: true, + filename: url, + }); + + return { format, source: transformedSource }; +} +``` +
+ +### `https` loader + +
+`https.mjs` + +```js +import { get } from 'node:https'; + +const mimeTypeToFormat = new Map([ + ['application/node', 'commonjs'], + ['application/javascript', 'module'], + ['text/javascript', 'module'], + ['application/json', 'json'], + ['application/wasm', 'wasm'], + ['text/coffeescript', 'coffeescript'], + // … +]); + +export async function load( + url, + context, + next, // In this example, `next` is Node’s default `load` +) { + if (!url.startsWith('https://')) { // Skip this hook for non-https imports + return next(url, context); + } + + return new Promise(function loadHttpsSource(resolve, reject) { + get(url, function getHttpsSource(res) { + const format = mimeTypeToFormat.get(res.headers['content-type']); + let source = ''; + res.on('data', (chunk) => source += chunk); + res.on('end', () => resolve({ format, source })); + res.on('error', reject); + }).on('error', (err) => reject(err)); + }); +} +``` +
diff --git a/doc/design/proposal-chaining-recursive.md b/doc/design/proposal-chaining-recursive.md deleted file mode 100644 index b4b2d7f..0000000 --- a/doc/design/proposal-chaining-recursive.md +++ /dev/null @@ -1,130 +0,0 @@ -# Chaining Hooks “Recursive” Design - -## Chaining `resolve` hooks - -Say you had a chain of three loaders, `unpkg`, `http-to-https`, `cache-buster`: - -1. The `unpkg` loader resolves a specifier `foo` to an URL `http://unpkg.com/foo`. - -2. The `http-to-https` loader rewrites that URL to `https://unpkg.com/foo`. - -3. The `cache-buster` that takes the URL and adds a timestamp to the end, so like `https://unpkg.com/foo?ts=1234567890`. - -In the new loaders design, these three loaders could be implemented as follows: - -### `unpkg` loader - -```js -export async function resolve(specifier, context, next) { // next is Node’s resolve - if (isBareSpecifier(specifier)) { - return `http://unpkg.com/${specifier}`; - } - return next(specifier, context); -} -``` - -### `http-to-https` loader - -```js -export async function resolve(specifier, context, next) { // next is the unpkg loader’s resolve - const result = await next(specifier, context); - if (result.url.startsWith('http://')) { - result.url = `https${result.url.slice('http'.length)}`; - } - return result; -} -``` - -### `cache-buster` loader - -```js -export async function resolve(specifier, context, next) { // next is the http-to-https loader’s resolve - const result = await next(specifier, context); - if (supportsQueryString(result.url)) { // exclude data: & friends - // TODO: do this properly in case the URL already has a query string - result.url += `?ts=${Date.now()}`; - } - return result; -} -``` - -The hook functions nest: each one always just returns a string, like Node’s `resolve`, and the chaining happens as a result of calling `next`; and if a hook doesn’t call `next`, the chain short-circuits. The API would be `node --loader unpkg --loader http-to-https --loader cache-buster`, following the pattern set by `--require`. - -## Chaining `load` hooks - -Chaining `load` hooks would be similar to chaining `resolve` hooks, though slightly more complicated in that instead of returning a single string, each `load` hook returns an object `{ format, source }` where `source` is the loaded module’s source code/contents and `format` is the name of one of Node’s ESM loader’s [“translators”](https://github.com/nodejs/node/blob/master/lib/internal/modules/esm/translators.js): `commonjs`, `module`, `builtin` (a Node internal module like `fs`), `json` (with `--experimental-json-modules`) or `wasm` (with `--experimental-wasm-modules`). - -Currently, Node’s internal ESM loader throws an error on unknown file types: `import('file.javascript')` throws, even if the contents of `file.javascript` are perfectly acceptable JavaScript. This error happens during Node’s internal `resolve` when it encounters a file extension it doesn’t recognize; hence the current [CoffeeScript loader example](https://nodejs.org/api/esm.html#esm_transpiler_loader) has lots of code to tell Node to allow CoffeeScript file extensions. We should move this validation check to be after the format is determined, which is one of the return values of `load`; so basically, it’s the responsibility of `load` to return a `format` that Node recognizes. Node’s internal `load` doesn’t know to resolve a URL ending in `.coffee` to `module`, so Node would continue to error like it does now; but the CoffeeScript loader under this new design no longer needs to hook into `resolve` at all, since it can determine the format of CoffeeScript files within `load`. In code: - -### `coffeescript` loader - -```js -import CoffeeScript from 'coffeescript'; - -// CoffeeScript files end in .coffee, .litcoffee or .coffee.md -const extensionsRegex = /\.coffee$|\.litcoffee$|\.coffee\.md$/; - -export async function load(url, context, next) { - const result = await next(url, context); - - // The first check is technically not needed but ensures that - // we don’t try to compile things that already _are_ compiled. - if (result.format === undefined && extensionsRegex.test(url)) { - // For simplicity, all CoffeeScript URLs are ES modules. - const format = 'module'; - const source = CoffeeScript.compile(result.source, { bare: true }); - return {format, source}; - } - return result; -} -``` - -And the other example loader in the docs, to allow `import` of `https://` URLs, would similarly only need a `load` hook: - -### `https` loader - -```js -import { get } from 'https'; - -export async function load(url, context, next) { - if (url.startsWith('https://')) { - let format; // default: format is undefined - const source = await new Promise((resolve, reject) => { - get(url, (res) => { - // Determine the format from the MIME type of the response - switch (res.headers['content-type']) { - case 'application/javascript': - case 'text/javascript': // etc. - format = 'module'; - break; - case 'application/node': - case 'application/vnd.node.node': - format = 'commonjs'; - break; - case 'application/json': - format = 'json'; - break; - // etc. - } - - let data = ''; - res.on('data', (chunk) => data += chunk); - res.on('end', () => resolve({ source: data })); - }).on('error', (err) => reject(err)); - }); - return {format, source}; - } - - return next(url, context); -} -``` - -If these two loaders are used together, where the `coffeescript` loader’s `next` is the `https` loader’s hook and `https` loader’s `next` is Node’s native hook, then for a URL like `https://example.com/module.coffee`: - -1. The `https` loader would load the source over the network, but return `format: undefined`, assuming the server supplied a correct `Content-Type` header like `application/vnd.coffeescript` which our `https` loader doesn’t recognize. - -2. The `coffeescript` loader would get that `{ source, format: undefined }` early on from its call to `next`, and set `format: 'module'` based on the `.coffee` at the end of the URL. It would also transpile the source into JavaScript. It then returns `{ format: 'module', source }` where `source` is runnable JavaScript rather than the original CoffeeScript. - -## Chaining `globalPreload` hooks - -For now, we think that this wouldn’t be chained the way `resolve` and `load` would be. This hook would just be called sequentially for each registered loader, in the same order as the loaders themselves are registered. If this is insufficient, for example for instrumentation use cases, we can discuss and potentially change this to follow the chaining style of `load`.