From ed0fab9ca47dbd23f85d6cd3296d3587d912b0ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Nison?= Date: Thu, 10 Feb 2022 12:48:28 +0100 Subject: [PATCH 1/6] Adds the ambient-loader proposal --- doc/design/proposal-ambient-loaders.md | 43 ++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 doc/design/proposal-ambient-loaders.md diff --git a/doc/design/proposal-ambient-loaders.md b/doc/design/proposal-ambient-loaders.md new file mode 100644 index 0000000..63ad821 --- /dev/null +++ b/doc/design/proposal-ambient-loaders.md @@ -0,0 +1,43 @@ +# Ambient Loaders Design + +## Problem + +Loaders let side logic be applied when resolving import statements. However, this only applies to the main application: loaders themselves aren’t currently affected by any other loaders. As a result, 3rd-party loaders cannot be injected if accessing them requires going through another loader. For instance, the following doesn’t work: + +``` +node --loader pnp --loader ts-node ./my-tool.mts +``` + +Indeed, importing `ts-node` would require the `pnp` loader to contribute to the resolution, but it’s not what happens today: both `pnp` and `ts-node` are resolved by the default Node resolver, preventing `ts-node` from resolving. + +The reason for that is an attempt to keep loaders as isolated as possible, to allow followup improvements like reusing them across workers or scaling them up. If all loaders were influenced by prior loaders, they’d effectively coalesce into a single one, which would prevent such work. Still, the problem remains. + +## Proposal + +If loaders cannot generally influence each other, we could have a subset of them do. Indeed, not all loaders affect the resolution so much that they are a requirement to followup loaders. We could have two levels of loaders: + +- Ambient loaders would be defined via the `--ambient-loader ` flag. They would be loaded sequentially, and would affect the resolution of all followup loaders. + +- Regular loaders would be defined via the `--loader ` flag. They would be loaded in parallel (at least conceptually). Because they’d only be loaded after the ambient loaders have finished evaluating, their resolution would be affected by ambient loaders. + +## Example + +Let’s imagine we have the following loaders: + +- pnp, which adds support for the Plug’n’Play resolution +- zip, which adds zip file support to the `fs` module +- yaml, which adds yaml file support to `import` +- coffeescript, which adds coffeescript file support to `import` + +The command line would end up like this (in practice `--ambient-loader` would probably be passed via `NODE_OPTIONS` rather than directly on the command line): + +``` +node --ambient-loader zip + --ambient-loader pnp + --loader yaml + --loader coffeescript + ./path/to/script.mjs +``` + +When resolving `coffeescript`, we’d go into the `pnp` loader (but not `coffeescript`, which isn’t an ambient loader). The `pnp` loader itself would have been loaded after the `zip` loader ran, causing its `fs` import to be resolved to a custom module adding zip support to `fs` rather than the usual builtin module. + From 96ecc5e9be71effabea8b27a66128fd52562ed06 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Nison?= Date: Thu, 10 Feb 2022 12:51:46 +0100 Subject: [PATCH 2/6] Update proposal-ambient-loaders.md --- doc/design/proposal-ambient-loaders.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/design/proposal-ambient-loaders.md b/doc/design/proposal-ambient-loaders.md index 63ad821..272f243 100644 --- a/doc/design/proposal-ambient-loaders.md +++ b/doc/design/proposal-ambient-loaders.md @@ -24,11 +24,13 @@ If loaders cannot generally influence each other, we could have a subset of them Let’s imagine we have the following loaders: -- pnp, which adds support for the Plug’n’Play resolution - zip, which adds zip file support to the `fs` module +- pnp, which adds support for the Plug’n’Play resolution - yaml, which adds yaml file support to `import` - coffeescript, which adds coffeescript file support to `import` +The first two are critical to a successful resolution: without `zip`, PnP wouldn't be able to load files from the zip archives. Without `pnp`, Node would look for the `yaml` and `coffeescript` loaders into a `node_modules` folder, which would fail. On the other hand, `yaml` and `coffeescript` don't depend on each other. + The command line would end up like this (in practice `--ambient-loader` would probably be passed via `NODE_OPTIONS` rather than directly on the command line): ``` From 706ea20528c2ede31f2cc7d644cee54bac39fd67 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Nison?= Date: Tue, 15 Feb 2022 18:23:36 +0100 Subject: [PATCH 3/6] Update doc/design/proposal-ambient-loaders.md Co-authored-by: Geoffrey Booth <456802+GeoffreyBooth@users.noreply.github.com> --- doc/design/proposal-ambient-loaders.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/proposal-ambient-loaders.md b/doc/design/proposal-ambient-loaders.md index 272f243..cd99cbe 100644 --- a/doc/design/proposal-ambient-loaders.md +++ b/doc/design/proposal-ambient-loaders.md @@ -16,7 +16,7 @@ The reason for that is an attempt to keep loaders as isolated as possible, to al If loaders cannot generally influence each other, we could have a subset of them do. Indeed, not all loaders affect the resolution so much that they are a requirement to followup loaders. We could have two levels of loaders: -- Ambient loaders would be defined via the `--ambient-loader ` flag. They would be loaded sequentially, and would affect the resolution of all followup loaders. +- Ambient loaders would be defined via the `--ambient-loader ` flag. They would be loaded sequentially and would affect the resolution of all subsequent ambient and regular loaders. - Regular loaders would be defined via the `--loader ` flag. They would be loaded in parallel (at least conceptually). Because they’d only be loaded after the ambient loaders have finished evaluating, their resolution would be affected by ambient loaders. From f403fd60b3c3d857c5f9d681e4010a9572bcd992 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Nison?= Date: Tue, 15 Feb 2022 18:26:53 +0100 Subject: [PATCH 4/6] Update proposal-ambient-loaders.md --- doc/design/proposal-ambient-loaders.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/design/proposal-ambient-loaders.md b/doc/design/proposal-ambient-loaders.md index cd99cbe..1ef027e 100644 --- a/doc/design/proposal-ambient-loaders.md +++ b/doc/design/proposal-ambient-loaders.md @@ -2,13 +2,15 @@ ## Problem -Loaders let side logic be applied when resolving import statements. However, this only applies to the main application: loaders themselves aren’t currently affected by any other loaders. As a result, 3rd-party loaders cannot be injected if accessing them requires going through another loader. For instance, the following doesn’t work: +Loaders let side logic be applied when resolving import statements. However, this only applies to the main application: loaders themselves aren’t currently affected by any other loaders. As a result, 3rd-party loaders cannot be injected if accessing them requires going through another loader. + +For instance, imagining a loader that would allow modules to be loaded from a `node_modules.zip` file (instead of the typical folder), the following wouldn’t work: ``` -node --loader pnp --loader ts-node ./my-tool.mts +node --loader zip --loader ts-node ./my-tool.mts ``` -Indeed, importing `ts-node` would require the `pnp` loader to contribute to the resolution, but it’s not what happens today: both `pnp` and `ts-node` are resolved by the default Node resolver, preventing `ts-node` from resolving. +Indeed, importing `ts-node` would require the `zip` loader to contribute to the resolution (so that the final path loaded is `$PROJECT/node_modules.zip/ts-node` instead of `$PROJECT/node_modules/ts-node`, which doesn't exist), but it’s not what happens today: both `zip` and `ts-node` are resolved by the default Node resolver, preventing `ts-node` from resolving. The reason for that is an attempt to keep loaders as isolated as possible, to allow followup improvements like reusing them across workers or scaling them up. If all loaders were influenced by prior loaders, they’d effectively coalesce into a single one, which would prevent such work. Still, the problem remains. From 630173d1fe5bfac5784a78cbd260aca3c56b5ce1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Nison?= Date: Mon, 21 Mar 2022 15:23:20 +0100 Subject: [PATCH 5/6] Adds requested tidbits --- doc/design/proposal-ambient-loaders.md | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/doc/design/proposal-ambient-loaders.md b/doc/design/proposal-ambient-loaders.md index 1ef027e..66377e3 100644 --- a/doc/design/proposal-ambient-loaders.md +++ b/doc/design/proposal-ambient-loaders.md @@ -22,12 +22,30 @@ If loaders cannot generally influence each other, we could have a subset of them - Regular loaders would be defined via the `--loader ` flag. They would be loaded in parallel (at least conceptually). Because they’d only be loaded after the ambient loaders have finished evaluating, their resolution would be affected by ambient loaders. -## Example +## Simple Example + +Let's imagine we have the following loaders: + +- ts, which adds support for TS files +- my-custom-loader, which is written in TypeScript + +Without the TS loader, we wouldn't be able to even load the following custom loader. But by marking it as an ambiant loader, Node will make sure it'll be taken into account when resolving (and loading) my-custom-loader. + + +The command line would end up like this: + +``` +node --ambient-loader ts + --loader my-custom-loader + ./path/to/script.mjs +``` + +## More Contrived Example Let’s imagine we have the following loaders: - zip, which adds zip file support to the `fs` module -- pnp, which adds support for the Plug’n’Play resolution +- pnp, which adds support for the Plug’n’Play resolution ([more details](https://yarnpkg.com/features/pnp)) - yaml, which adds yaml file support to `import` - coffeescript, which adds coffeescript file support to `import` From f4c816b79530f505342f6ddc971e6ea568f959d2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Nison?= Date: Fri, 25 Mar 2022 13:22:29 +0100 Subject: [PATCH 6/6] Update doc/design/proposal-ambient-loaders.md Co-authored-by: Geoffrey Booth <456802+GeoffreyBooth@users.noreply.github.com> --- doc/design/proposal-ambient-loaders.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/doc/design/proposal-ambient-loaders.md b/doc/design/proposal-ambient-loaders.md index 66377e3..33eb016 100644 --- a/doc/design/proposal-ambient-loaders.md +++ b/doc/design/proposal-ambient-loaders.md @@ -26,17 +26,16 @@ If loaders cannot generally influence each other, we could have a subset of them Let's imagine we have the following loaders: -- ts, which adds support for TS files -- my-custom-loader, which is written in TypeScript - -Without the TS loader, we wouldn't be able to even load the following custom loader. But by marking it as an ambiant loader, Node will make sure it'll be taken into account when resolving (and loading) my-custom-loader. +- `ts`, which adds support for TypeScript files +- `my-custom-loader`, which is written in TypeScript +Without the `ts` loader, we wouldn’t be able to even load the following custom loader. But by marking it as an ambient loader, Node will make sure it’ll be taken into account when resolving (and loading) `my-custom-loader`. The command line would end up like this: ``` -node --ambient-loader ts - --loader my-custom-loader +node --ambient-loader ts \ + --loader my-custom-loader \ ./path/to/script.mjs ```