-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal - named/mapped base URLs #3161
Comments
The problem with all of these bare specifier proposals is, when it comes to modules, all specifiers are contextual. The meaning of This works fine if your usage is restrained to only your own code (ie. you aren't using libraries), but once you bring in outside code you have to deal with the meaning of their specifiers. I think the only real solution to this problem is some sort of path resolution api. I see two primary use cases for bare specifiers on the web:
Both of these use cases are well solved today by creating a wrapper module. For example if you use the export * from 'https://example.cdn/alpha@1.0.0/dist.js'; And then have all usage of alpha point to this |
Thanks @matthewp for taking the time to give some feedback.
I did try to address this in my proposal. In the example in the description above, when a dependency (bravo) references a dependency (alpha) within a script or stylesheet it must be translated to the context of bravo first. So import "alpha" -> "bravo/alpha" -> "https://example.com/example.js". I also have one case like this in my example repo here. This doesn't seem all that difficult to manage to me. The context already has to retrieve the document base url so this would just be an extra check to see if the url is a bare specifier and is mapped and follow the tree. The only potential issue I can see with this approach is two modules depending on the same module from two different locations (i.e. cdn) which will cause the module to load twice. This could be solved though with a lock file. {
"alpha": "https://example.cdn/alpha@1.0.0/dist.js",
"bravo": "https://example.cdn/bravo@1.0.0",
"delta": "https://example.cdn/delta@1.0.0",
"@dependencies": {
"charlie": "https://example.cdn/charlie@1.0.0/basemap.json"
},
"@overrides": {
"charlie/delta": "https://example.cdn/delta@1.0.0",
}
} Or something similar.
That's kind of what I am describing. It's almost like a virtual directory structure or symbolic linking. |
Proposed ways of mapping would work for particular cases and most likely would not for others. That way hardcoded values either in import "url" or import(url) or <link rel=import href=url could be handled as locally as on individual component sub-level. One of versions of such API could be an event kind of subscription and bubbling from current module up to caller level. The higher level could override path mapping. Not sure about stopPropagation though. |
Could you provide an example or two of where this proposal would not work?
This proposal offers a solution for this as well. Perhaps I am failing to explain it correctly. Let me try to give a simpler example My basemap.json {
"@dependency": {
"alpha": "https://example.cdn/alpha/basemap.json"
}
} The alpha dependency basemap.json {
"bravo": "https://example.cdn/bravo/dist.js"
} And now from within a script in alpha somwhere import bravo from 'bravo'; 'bravo' is actually mapped from the alpha level. When the context retrieves the base url from the root document it will just follow the tree for the correct map. It will see that the context is within the alpha dep sub tree and map bravo to "https://example.cdn/bravo/dist.js" This assumes that module developers will accept the standard and create the proposed basemap.json (or whatever it is called). However, there is a way to map this from the root so that developers that consume older modules don't have to wait for the module to be updated.
I can also envision a path to a completely automated basemap.json when using package managers like npm to install modules that doesn't require the dependencies to follow this standard. |
@shannon , After some thinking I found that mapping API and data format for defining the rules are orthogonal to each other. And seems it worth to make an another separate proposal for URL mapping API. Your json mapping proposal is good for many cases and perhaps could be considered as default implementation for rules definitions. There are other possibilities though to be discussed.
There are virtually unlimited incompatible or partially incomplete variations. Some:
The cases are quite exposed and reflected in AMD configurations for packages, aliases, etc.
But I would say the trouble is not format (json vs regex vs JS lambdas...) but rather taking the side. Standard IMO should accommodate the needs but not limit. In your format I see a nice default implementation flexible enough for many but not all cases. I easily could imagine the XML and XSLT transformation rules to serve "universally" all cases, but would it fit all? Absolutely not. The API should cover the functional side allowing various implementations to compete and let the user win :) PS. Forgot another significant use case: bundling within same file. It is more sophisticated as most likely would require not just change the URL mapping but loading as well. |
Could not stop myself. What about the fallback URL mapping in case of CDN failure? Good for progressive apps with original resources (js, HTML, ...) kept locally. In runtime resources located on corporate CDN, but if not available fall back to public CDN and finally back to local. |
I suppose this proposal could allow regex maps but this seems to me like a very confusing way to structure your app. If this is a result of a package manager or build step than the base map file can be generated automatically and verbosity becomes less of an issue. However, I believe the two examples in your aliases config could be solved by this proposal as well: {
"vaadin-": "<vaaidn_root>/vaadin-",
"iron-": "<iron_root>/iron-"
}
This could easily be handled by writing the base tag with a script and conditionally loading a different named href or map. <script type="text/javascript">
document.write(`<base name="link-import" href="${ isLocal ? '/components/link-import/link-import.js' : '../../link-import.js'}" />`);
//or
document.write(`<base map="${ isLocal ? 'local-basemap.json' : 'cdn-basemap.json'}" />`);
</script> This same approach could be used for any fallback URLs. The biggest disadvantage I can see with using a functional API is that the tree can not be statically analyzed, which if I'm not mistaken was a big driver in the ES Module spec as it is. |
@sashafirsov I think I misunderstood your fallback URL comment. |
@shannon , Initially I meant the runtime switch based on some state which is set before URL resolver use. For example first synchronous script resolved to host and host is not available. The mapping from this host ( say defined in this JS ) has no sense as not available. The alternative location, perhaps with own mapping should be tried. At such scenario I thought of 3 different APIs: URL resolver, Load lifecycle callbacks, loader. But you given an idea of alternative mapping descriptors which will identify the sequence of attempts for different mappings. It is quite attractive by its simplicity and fit into current logic, just define extra mapping entry and it will be applied in case the previous failure. Unfortunately such hardcoded sequence could not be changed in runtime except of document.write() or on html level by build or backend. Which does not look like a good suggestion in w3c scope. The API layers above could handle gracefully via JS. |
@shannon ,
Here I see 2 problems:
That could be an answer. But in this case the key will be a regex and value should have a regEx replacement convention with group indexing. What about using mask * and ? as used in file system cli with same replacement rules? The RegEx seems to be a decent default for static mapping. Other formats could be custom with reference implementations using API. |
The API does not replace the declarative proposal, hence the static analysis could be done as is. If you recall the history immediately after ES6 modules introduced the multiple builders appeared to cover the lack or module URL resolvers. Which literally replace ID with path. Having the resolver API would make such transformations unnecessary (still handy for other reasons though) . By the way, for your proposal you would need a shim for legacy browsers. Why not make it via API layers I mentioned? It will be serving default mapping according to proposal and still allow the custom mappings in whatever developer decides format. |
Yea this sounds to me like it solves some different problems. I'd be happy to participate.
Yea this would need to be a document.write or some other scripted append. If we support recursion, an easy way to manage this would be to create a map for this root first. <script type="text/javascript">
document.write(`<base name="vaaidn_root" href="${VAADN_ROOT}" />`);
document.write(`<base name="iron_root" href="${IRON_ROOT}" />`);
</script>
<base map="basemap.json" /> basemap.json {
"vaadin-": "vaaidn_root/vaadin-",
"iron-": "iron_root/iron-"
}
With this proposal, the key as always just a prefix. It's a base path. A distinction between prefix and solid string is unnecessary. {
"foo": "https://example.cdn/foo"
} import foo from 'foo'; //translates to "https://example.cdn/foo" this is technically valid because browsers don't care about extensions
import foo from 'foo.js'; //translates to "https://example.cdn/foo.js"
import foobar from 'foo-bar.js'; //translates to "https://example.cdn/foo-bar.js"
import foobar from 'foo/bar.js': //translates to "https://example.cdn/foo/bar.js" Just replace Suffixes and substitution from the base map json file would not be supported by this proposal. This sounds out of scope of what this proposal is trying to solve. It's clear you are suggesting a more dynamic URL resolving solution than what I am proposing but I just can't see a clear use case for changing the mapping at runtime.
I'm not even sure that this can be shimmed as a bare import specifier will currently throw. |
This proposal seems reasonable, but I'm not sure if anyone is representing NodeJS/npm here ... if beside the Web we could get Node/npm compatibility we'd surely all move to a better direction. cc @MylesBorins |
@WebReflection I had hoped that this proposal would bring the web more inline with the way the node resolver works. I believe it's close enough that it should be straightforward to convert a package-lock.json file to a basemap.json file. I would expect that inferring values from package-lock.json I would end up with a basemap.json file that looks something like this: {
"example/": "./node_modules/example/",
"example": "./node_modules/example/main.js"
} So when I import example import example from 'example'; I get the file designated as the main in example/package.json. And when I import a relative file to the module it resolves accordingly import subexample from 'example/sub/example.js'; I say package-lock.json and not package.json because:
Of course, package-lock.json would need the main property for this to work and I don't know how much resistance that would get. In either case there are more ways to get the structure and map it out. So as an application developer I can use npm to install my dependencies and generate a basemap.json that should make it work for the web. The only part that would be an issue is how the node resolver adds the file extension. This would be a problem for using libraries that leave the extension off with relative paths. In theory you could create a map that contains every file with and without extension so you could leave it off. I don't know how this would impact performance though. I'm not really sure what the community is thinking about this problem. If the web server supports the right default extension it might not even be a problem. |
absolutely. If you ignore the fancy nested resolution bits, this already works with CJS or ESM modules and developer could start publishing pre-bundled modules to simplify Web interchangeability, when needed. this would also solve entry points for full valid ESM modules based on relative/absolute paths too, even if resolving within those remote modules mapped names might be more tricky.
tools are there to help. You might need zero changes in the current system to make it work out of the box.
The right default extension for JavaScript files is, ad hopefully will always be, But again, tools can easily pre-solve all of this so ... not an issue, really. The only reason I've asked about node/npm folks is that maybe there are hidden caveats or suggestions not considered in here that could benefit both platforms. I can see Thanks. |
Nice, elegant proposal. Its mappings of URL prefixes and base paths reminds me of the already-specced JSON-LD and its approach to URL prefixing. The keys in JSON-LD data are interpreted as compact URLs, which are literally URLs abbreviated with prefixes separated by colons, like one might see in namespaced XML. The W3C formalized this general concept with the funky name “CURIEs”.) I’m not actually proposing this seriously, but it was amusing for me to think about how some of the examples above might adopt a syntax resembling JSON-LD’s contexts, so that:
…with this mapping:
…would become something like this:
I’m not really making a serious proposal for this; I’m just thinking about how to match prior art, for fun. I do want to caution that, no matter the proposal, it’s important that any URL-resolution process on the web not privilege any special file extension, including After all, WebAssembly is already being deployed. Once integration with ES Modules occurs, web applications might be replace JS modules with WASM modules without changing the URL, e.g., using HTTP content negotiation on The original proposal is indeed forward compatible, since it just uses standard URL resolution against base paths—but I just wanted to make sure that this issue remembered, since URL opacity is a fundamental but often-forgotten aspect of the web’s architecture. On the whole, this proposal itself is simple and elegant, though I’d love to read about alternatives to than this and those in #2640. As Denicola said, this problem is very tricky, and it’s probably really easy to get trapped into an AppCache-like situation. |
to be clear, I think developers just need to understand that ESM is not CJS and they should use fully qualified names that work both on file system and Web (that includes wasm, css, js, whatever) Pure ESM (no, no what tools addicted write, I mean the ESM that works without tools) already works well on both client and server (JSC, SpiderMonkey) and this proposal is perfect to address libraries by name too. On node, the node_modules/package.json will help resolving them, on My only concern is that it's not fully clear, if not using toolings, how am I supposed to The solution here makes a no-brainer to address the first Again, easy to solve via tooling, but not super-clear for developers that would maybe like to play around first. |
@js-choi I agree, that's why it could be left up to the web servers to help here. There's no reason why if an extension is left out down in the tree somewhere the web server couldn't search a few different default extensions to find the correct file. This is already pretty standard across the web with .html, .php, etc. The browser won't have to do anything to add extensions. And absolute extensions could be added to the map during the development phase using tooling to avoid any performance hit on the web server or if the web server does not support this.
@WebReflection so without tooling, yes, Example {
"a/b/": "./node_modules/b/",
"a/b": "./node_modules/b/main.js",
"a/": "./node_modules/a/",
"a": "./node_modules/a/main.js"
} So, the lookup logic would be something like
The rules here would need to be fully fleshed out for edge cases but that's the gist of it. It's certainly not the simplest thing to think about but I think most people that are looking to use something like this would understand it. It's no more complicated than CSS specificity :-) My biggest personal goal here was to actually remove tooling so if I am just writing pure ESM, which I try to do these days, I can include a basemap.json and move on. In reality if I want to use third party libraries I have two major options. Use a CDN or use a package manager. Things become tricky when using a CDN with a deep dependency tree. If a library in a CDN depends on the same package as my application but it's located somewhere else, the browser will load it twice. Even with this proposal. We don't want this. So using a package manager becomes attractive again. And with my "tooling" limited to just something like I dream someday I can type something like Edit: forgot to @js-choi |
but Anyway, I've got the answer, thanks. |
@WebReflection Actually, I hadn't thought about it just figuring it out automatically from your import statements but yea that should theoretically be possible for static imports. Thanks for the feedback. |
I think it's best to close this as work has shifted to https://github.com/WICG/import-maps :) |
I've been thinking a lot about the ES Module bare import specifier and what that it is going to look like in the future. I've seen a few proposals but I thought I would offer my own.
A lot of this comes from the discussions in the following two issues:
#2640
tc39/proposal-dynamic-import#52
I hope this is the right place for something like this. I've never written a proposal like this so if this belongs somewhere else please just let me know.
Proposal - named/mapped base URLs
This proposal is to add two attributes to the
<base>
tag calledname
andmap
.What are we trying to solve?
Essentially, we want to be able to specify a bare name in a URL (e.g. import specifiers) and have that name translated to a real URL on request.
Translates to:
Having said that, I won't be mentioning ES Modules for a large portion of this proposal because the specifics of ES Modules are largely irrelavant to the proposed change.
Constraints:
Working with all resource types
Because this needs to work for all resource types I propose that we extend the
<base>
tag to provide this functionality. The base tag already performs similar translations and with a few simple modifications we can leverage that here.I propose that if a base tag has a
name
attribute it creates a new concept calledNamedBaseURL
.This works a lot like the standard base tag except it only remaps relative URLs that don't begin with
./
or../
and starts with with the name from aNamedBaseURL
.Translates to:
In practice, if you want to name a third party bundled resource you could just point to a specific resource instead of treating it like a base url.
Translates to:
Working with a deep dependency tree
Once we have the concept of a
NamedBaseURL
, I propose we add a second base tag attribute calledmap
. Themap
attribute points to a json file with a key value pair of names and URLs. This in turn creates eachNamedBaseURL
accordingly. A relative URL value is calculated relative to the location of the map json file.basemap.json
To create a deeper dependency that will have it's own
NamedBaseURL
map we can simply use both attributesname
andmap
. This concatentates the name with the keys from the specified map.alpha-basemap.json
Then it becomes fairly straightforward to create a deep dependency tree if we just add a special key called
@dependencies
. It holds a key value pair for additional maps which are fetched and creates newNamedBaseURL
objects with bothname
andmap
accordingly.bravo-basemap.json
Could be equivalent to
For deep depencencies the URL translation would need to account for relative named base urls from the context of the resource. In other words, when
alpha
is requested from within the context ofbravo
(from inside a script or css) it should translate tobravo/alpha
before ultimately translating to the full URL. However, I think this would be pretty straightforward and simple to understand.Locking for integrity
Since a dependency potentially isn't bundled or tarballed like npm we would then need a checksum/hash for each individual sub resource. It could be mapped accordingly with an
@integerity
property. Of course you would need to know all the resources for the dependency before hand. Internally, the host environment would just check if a resource has a mapped hash and throw an error if the hash does not match.basemap.lock.json
Example
See https://github.com/shannon/named-base-url-proposal/tree/master/example for a quick example of how it would all work together
The text was updated successfully, but these errors were encountered: