A request data -> request hook (e.g. module specifiers -> module request)

[annevk]

(Note: OP edited by @domenic to add some more context)

A request that often comes up in the context of modules, but is also applicable to other resources, is the ability to map resource specifiers (e.g. import "x" or <img src="x">) to different URLs than the one that would be naively computed.

@guybedford closed #2547 where we discussed possible alternatives to the Loader API and what role service workers could play in them. The TL;DR is that we first want more experimentation to see what folks come up with.

However, given that there's continued interest in this area and I think such experimentation will show the need for a hook of sorts to make solutions less of a hack, I'm opening this issue.

Please read #2547 for more background about the problem (for modules) and discussions toward a solution (for requests in general). This thread proposes a specific direction, although we can also use it to discuss other solutions to the problem.

The rough idea is that in the same task we dispatch the service worker fetch event, we first dispatch another event to transform a set of requests inputs into a request. That way you can create your own identifiers for resources and let the service worker take care of mapping those identifiers to URL, integrity, referrer, etc. information (a request) that is then used to fetch the resource during the fetch event.

How this works in detail is a little tricky. If you have something like <img src=identifier> on https://example.com/, <img>.src will return https://example.com/identifier whereas that might not be the mapping the service worker has in mind. But that is probably fine as the service worker can already do whatever it wants so <img>.src is not trustworthy information. All we need to make sure of is that it gets identifier as "raw data" as well.

cc @jakearchibald @dherman

[jakearchibald]

One use-case I have came from @domenic:

// main.1.0.0.js

import {something} from './tiny-library.1.0.0.js';

// …LOADS OF CODE…

But uh oh, there's a bug in tiny-library! Thankfully it's fixed in 1.0.1. Unfortunately, in order to start using 1.0.1 you have to alter main.1.0.0.js, bumping it to main.1.0.1.js etc etc. For a tiny change you invalidate loads of script.

You can workaround this with the service worker, but you'll miss it on the first load. It'd be nice to be able to do something like:

<script type="module-map">
{
  "main": "./main.1.0.0.js",
  "tiny-library": "./tiny-library.1.0.1.js"
}
</script>
<script type="module">
import "main";
</script>

Where main.1.0.0.js is:

import {something} from 'tiny-library';

// …LOADS OF CODE…

Now you can bump the version of particular modules in the mapping, without invalidating the whole branch.

[domenic]

Any solution here should not be module specific.

[jakearchibald]

It would have to be declarative though, right? Pre-parsers would need to execute it.

[domenic]

Yeah, which makes this design process very tricky (easy to end up with app cache). That's why I think we need a lot more prototyping and experimentation and use case gathering, even if prototypes defeat preload scanners.

[bmeck]

Is there an up to date list of design requirements for this? I am working on hooks for Node, and trying to get a hold of compatibility path (I assume it will require a build step most likely).

[annevk]

Maybe someone from @whatwg/loader has up-to-date insights, but I suspect there simply hasn't been much bandwidth to get this to move yet. And modules still not being widely deployed doesn't help with feeding requirements and use cases.

[bmeck]

@annevk some still seem to exist though, like the preparser requirement.

edit: this is of note since it was not in @whatwg/loader

[WebReflection]

OK, this discussion is fairly little, happy to chime in after seeing this issue closed as duplicate.

Use case

I think @jakearchibald already mentioned the most common use case of all, which is the reason we all use package.json and require by package name on the server.

Moreover, having a way to map tiny-library or any other as resource gives back roles to CDNs and HTTP2 requests.

Today the dynamic Web can hardly exist as a platform without bundlers. Current ES2015 browsers module implementation makes usage of these bundlers mandatory indeed. Nobody can load libraries in any reasonable way and make the same code portable between environments (transpiled, compiled, etc).

Simplicity

I think both me and Jake having same idea indicates it's obvious/simple to think about it, explain it, and also implement it.

Accordingly, since import is a JavaScript module specific mechanism, I don't see why we need an overengineered solution or nothing.

Any solution here should not be module specific.

Modules are the only thing that are frequently needed, differently from images or CSS, and used multiple times per each file, differently from anything else on the Web.

How come the most needed bit, which is the ability to load modules the way we've been doing for the last 10 years, is suddenly something every Web resource needs to do?

I don't see the use case for that.

Not only Service Workers

Currently Service Workers do their job only the second time the page is loaded.
AFAIK there's no way to use SW to serve the current, first time, JS dependencies on the same cache, so SW would not be a solution.

Mapping there is probably a no brainer, but everything would be broken without SW. Does it really have to be mandatory? So I loop back to Simplicity.

Alternative

My proposal (on a second though) was based on a static file on the server with a specific script type too.

<script type="module-namespaces" src="/mjs.json"></script>

call it module-map as Jake suggested, works for me. That very same file could be fetched via SW a part, and be executed as non-blocking, one off, resolution before executing the first module on the page (if any), 'cause modules are asynchronous anyway, and the mechanism to load/import them is also non-blocking.

It'd be just one extra request, if present, and it can instrument bundlers/loaders/polyfills to resolve dependencies in a better way upfront.

This is a module specific solution because AFAIK there are no other use cases that really need this solution so I'd say why not moving forward the easy way?

[WebReflection]

Summarizing here the pros on using a static json file since there's nothing else really interesting in the other thread at this point:

Pros

• security: the website owner is the only one responsible for such file, no CORS should ever be enabled
• statically valid: it is never possible to change this file, or the URL pointer, at runtime. You cannot create script of type module-map or module-namespaces
• polyfillable: this file can be used to instrument loaders/bundlers and somehow polytfill the described behavior/resolution for modules.
• smart cache: browsers might pre-load modules ahead of time when the network is not busy, and/or cache them when possible to have same resolution across different sites
• performance: HTTP2 and CDNs finally get their role back instead of being nullified by bundles
• portability: every module can point at a relative/absolute path like it is already for ES6 modules on browsers.

Specially last point ensures JS code can be shared and ported across different environments that uses different bundlers or different sources based on package.json.

mjs.json example

{
  "hyperhtml": "https://unpkg.com/hyperhtml@latest/min.mjs",
  "lodash": "https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.4/lodash.min.js",
  "site-lib": "/js/lib/site-lib/index.mjs"
}

mjs.json via web

<!doctype html>
<script type="module-map" src="/mjs.json"></script>
<script type="module">
import hyper from "hyperhtml";
addEventListener('load', () => {
  hyper(document.body)
    `<h1>Welcome in ${location.hostname}</h1>`;
}, {once: true});
</script>

[WebReflection]

after @guybedford comment on the other thread:

Add "integrity" and the dependencies list per module for preloading to this format, and I'd be happy :)

I don't think there's a need for a universal solution that covers both node and web.
These are two completely different platforms, with completely different libraries, for completely different needs.

NodeJS has no <script type="module"> to deal with at all. It has no Service Worker, it never comes with pre-bundled modules so that I don't see this proposal as a solution capable of solving dependencies list of mapped external resources for the following reasons:

• the Web comes bundled, specially libraries in CDNs
• if there is a known dependency, the /mjs.json file would be statically analyzable and eventually it can contain in itself dependencies needed by other libraries.

I have a concrete example right here of what this proposal would solve:

• I have a library exported as ES2015 module
• I have another library that would like to import the first one as dependency

Shipping twice the first library would be redundant. Using relative path would be not portable. Using bundlers would mean being incapable of serving library 2 from a CDN.

The solution is to let the environment solve the issue, simply importing library 1 on top.

NodeJS would know how to load and resolve that while on my webpage, my /mjs.json file would tell the browser where to find it.

Any tool could eventually pre-parse dependencies upfront per each needed module, and automate the creation of such mjs.json map file.

Accordingly, I see this thread as a generic NodeJS solution, but not necessarily a good fit for what the Web needs.

[WebReflection]

About being module specific

last thought on this:

Any solution here should not be module specific.

I don't understand how importing bootstrap could be distinguished between the JS module or the CSS resource, but if the mjs.json file I've mentioned is not seen as possible good solution for that, I wouldn't mind having it re-designed as such:

{
  "bootstrap": {
    "module": "https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js",
    "style": "https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
  }
}

The module is used when the import comes from <script type="module"> while style could be used when the @import comes from CSS.

Of course if there are better names for that, I wouldn't mind a change.

[medikoo]

@WebReflection how it's supposed to resolve dependencies of dependencies?
e.g. let's say code in your example imports bootstrap, so you provided /mjs.json as:

{
  "bootstrap": {
    "module": "https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"
  }
}

Now let's say bootstrap internally imports jquery.

Should your /mjs.json list it as well (so also all dependencies of dependencies)? (if so, how should we handled cases where A needs C at v1 but B needs C at v2?).
Or should it then resolve another /mjs.json from bootstrap site (https://maxcdn.bootstrapcdn.com/ ?)

Technically it needs to be lean and fast (as it's with bundling) for cases where we deal with 500+ of modules with deep nested dependencies, and sometimes containing same packages at different versions (that's reflection of real world setup of many today's applications).

[WebReflection]

@medikoo I've answered already at that:
#2640 (comment)

TL;DR YAGNI, libraries are deployed to CDNs already as bundle.
We don't want/need NodeJS capabilities on the Web, we just need to recognize it's a different paradigm!

[medikoo]

TL;DR YAGNI, libraries are deployed to CDNs already as bundle.
We don't want/need NodeJS capabilities on the Web.

Ok, so you see ESM purely as a modules format for bundles (or large modules)?

Or do you suggest that transpilation should be a mandatory step for web (when we deal with more complex applications e.g. built of 100+ of smaller modules), as we cannot bundle few ESM modules into one ESM without transpiling them into something else.

[WebReflection]

Ok, so you see ESM purely as a modules format for bundles (or large modules)?

No. relative and absolute paths are still fundamental for ESM. Here we are trying to solve external libraries dependencies. You don't want to dig into the "need to resolve also their inner dependencies" rabbit hole because that is not a real-world use case.

Libraries are published to CDNs already as bundle, this is a simple fact.

Or do you suggest that transpilation should be a mandatory step for web

No. I am actually promoting what's already there and what worked already. External dependencies are already bundled, nothing new, nothing different to learn, nothing to change, except exporting as .mjs or as ES2015 module.

However, your own code in your own site can use as many relative/absolute dependencies.

I am solving the only issue I have with the current ES2015: I cannot require an external library.
Everything else is YAGNI to me.

[WebReflection]

Moreover

The format I've proposed is compatible with any external dependency as flat tree. If you can statically analyze the file you can also statically analyze each package.json per each dependency and create a flat hierarchy of dependencies.

If this approach doesn't scale enough, we can eventually use the latest proposed format and simulate what package.json does.

{
  "bootstrap": {
    "module": "https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js",
    "style": "https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css",
    "dependencies": {
      "jquery": {
        "module": "https://maxcdn.bootstrapcdn.com/jquery/3.3.7/jquery.mjs"
      }
    }
  }
}

Every remote library can use relative paths that will be resolved through their remote location and every remote library can import its dependencies as defined in that file.

This would cover all scenarios

edit nested dependencies are actually a road to hell here .... so I am not in favor of my own alternative solution to dependencies

[WebReflection]

Last thought for @medikoo : you never want in production to load relative files because these will never be minified/optimized for the web, which is why I am insisting external dependencies bundled for production should not be a concern here. I would never import in production something that hasn't been optimized and AFAIK nobody imports relative files using import * from "./module.min.js" 🤷‍♂️

I also would love to keep it simple as much as possible instead of being stuck with this for years.

[medikoo]

Libraries are published to CDNs already as bundle, this is a simple fact.

I have a feeling that you're trying to send us back to times pre node/npm where we e.g. landed jquery in one script tag, in the other main.js script,where we configured some behaviors relying onjquery dependency.

The way modules are connected and resolved now in node.js/npm (with semver on board) env, goes far beyond and changed whole lot in a great way.

Applications I work with now are built of 700+ modules, where majority of that is external dependencies with it's dependencies etc.. Additionally 40% of that codebase is also run in Node.js env. It's pure CJS modules, bundled for browsers, and which work with no transpilation in both Node.js and browser environments efficiently.

I totally don't see how what you propose can replace setup (I described above) in a good way, at least it definitely doesn't empower ESM with CJS capabilities it lacks, but maybe my problem is that I assumed you try to solve exactly this issue, when it's not the case.

[WebReflection]

I have a feeling that you're trying to send us back to times pre node/npm

I am just bringing you back to reality. Every dependency we have on the web is bundled because:

• it needs to be minified in its entirety, not just its entry point
• accordingly, it is in the CDN already bundled

I can link the million libraries out there served through CDNs but I am fairly sure you understand what I am talking about, which is not jQuery.

It's pure CJS modules, bundled for browsers

Exactly. You are confirming what I am saying. Libraries/applications ships already bundled, because of the points I have already mentioned. You don't want to trigger 700+ network requests * 700 multiple dependencies on the Web, do you understand what I am saying?

You want to load a library published in the CDN, optimized, and free of dependencies resolutions, like it is already for every library on the Web, unless you bundle it.

This means it's not me bringing you back to bundled libraries, it's you stuck behind bundlers no matter what, because that's optimal for the web. The Web is not NodeJS, and it should never bury itself to have full CJS capabilities: that is not the Web use case, that is not what we need to improve modules sharing, IMO.

[medikoo]

it needs to be minified in its entirety, not just its entry point

I think you're focused on optimisations that target production environments, and forgot about other cases.

By no means practices as minification, transpilation etc. were in a past (or should be now) mandatory for normal web development (no matter whether app is really big and complex or we'll learning to build something).

Thing that transpilation step is required now with ESM is main reason behind JS fatigue we frequently read about. If that's not solved then ESM (in my personal opinion) is not worth consideration.

[WebReflection]

transpilation and ESM have nothing to do with each other ... indeed, you never want to transpile ESM on the Web, different story for NodeJS, where everyone trapped itself behind transpilers.

If transpilers were not so popular, people using import ... would've implicitly consider the imported module ESM, and people using require would've implicitly consider the imported module CJS.

Today we have everyone using ESM, transpiled to CJS, so that shenanigans like .mjs are needed.

The Web doesn't need that, and there are best practices on the Web since ever.
Libraries can, and shoud, be pre-bundled, because that's how you ship production.

For non production use cases, you can do whatever you want. You have tools, and you're using tools, and you'll always do that because that's the way to go, that's the way to create production bundles.

My proposal addresses this case, which is the only one that's relevant: production external dependencies.

Everything else can be solved by tooling and relative paths if you want, that's not what the Web need, just what lazy developers might use. Is that mandatory? Not for local usage, where you use rollup, webpack, browserify, you name it ... nobody cares.

On the web? You want to be sure you can use external dependencies and these are always served already bundled. This is how the efficient, production web, works.

[WebReflection]

Also ... we can agree to disagree, 'cause I've nothing else to say about it.

[bmeck]

@annevk do you still want to wait on userland for this?

[annevk]

Yeah, I think service workers being deployed everywhere will help drive more of a need for this. The other thing I'm interested in seeing is if https://wicg.github.io/origin-policy/ will work out. If that's successful maybe there's an opening for loading a small service worker before the main content as well, solving the scanner issue.

[bmeck]

@annevk with all the caveats of current limitations, I've thrown up https://bmeck.github.io/node-sw-compat-loader-test/dist/ as an example of what it takes to do this today. Extra network requests are plentiful and so are CPU draining source code transforms inside of service workers (doesn't seem to be a way to throw things into a worker). The proof of concept is minimal and could be expanded further to do things like code instrumentation instead of being focused on rewriting specifiers to other URLs but I wanted to leave that out for now.

[annevk]

Going to close this in favor of #4938.