Use unpkg as a SystemJS source

[sebastien]

It would be nice to use Unpkg as a universal source for packages when using dynamic loaders such as SystemJS. An ideal configuration would be:

SystemJS.config({paths:{"*":"//unpkg.com/*.js?as=umd"}})

Allowing the following forms:

require("jquery")
require("gl-matrix")
require("tgd")

Right now, this would not work for modules such as gl-matrix and tgd for the following reasons:

• gl-matrix exports the ES6 code by default, but has a dist version with an UMD build at https://unpkg.com/gl-matrix@2.3.2/dist/gl-matrix.js
• tgd exports the UMD build under the umd directory and uses umd/index.js as the main file, however the relative imports fail because the Unpkg URL is unpkg.com/tgd and not unpkg.com/tgd/umd

In order to better support SystemJS-like paths, Unpkg could:

• Add a as=umd parameter with the following semantics: look for an umd or dist directory, and use it as root.
• If an index.js is present in this directory, it should be served by default when the request URL path does not end with a /. For instance https://unpkg.com/tgd@0.0.4/umd/ and https://unpkg.com/tgd@0.0.4/umd now return the same, while it could https://unpkg.com/tgd@0.0.4/umd return the contents of https://unpkg.com/tgd@0.0.4/umd/index.js (or at least redirect to it).

The relatively simple changes would allow to bypass package management systems such as NPM or JSPM and use unpkg as an online, versioned, package source.

[mjackson]

I don't have a lot of experience w SystemJS, but just looking at the config you've got there, could you do something like this right now?

SystemJS.config({
  paths: {
    "gl-matrix": "//unpkg.com/dist/*.js",
    "tgd": "//unpkg.com/umd/*.js"
  }
})

[sebastien]

Yes, that might work, but then I would need to specify each dependency (and the dependencies's dependencies) in the the paths. With the solution outlined above, you just need to configure the catchall path and most require should succeed without additional configuration.

[mjackson]

What is the root problem here?

I see that in the tgd package you're using "main": "umd/index.js". So, technically if you have a require("tgd") that should just work. Are you trying to require other files deeper inside the tgd package? What do your require paths look like?

[mjackson]

Also, do you think the the browser package.json field could help here in some way? Sorry for so many questions. I'm just trying to understand the root of the problem before we talk about solutions.

[sebastien]

The root problem is as follows: we do not want to manually register the unpkg URLs for every single package (including their dependencies).

For instance:

• tgd contains pre-built files in the umd directory, as a result, the ideal generic SystemJS config path "*":"//unpkg.com/*.js" would not work. Instead, one has to specify "tgd": "//unpkg.com/tgd/umd/index.js"
• tgd transitively imports gl-matrix and performance-now. gl-matrix has a pre-built UMD file in dist. Like with tgd, we can't use the generic "*":"//unpkg.com/*.js", and instead need to add "gl-matrix": "//unpkg.com/glmatrix/dist/glmatrix.js"

And yes, taking hints for the browser field would probably help, although this is more a nice to have.

[sebastien]

Hi Michael! I was wondering if you needed some help to better formalize the feature, I could also give a hand for implementation.

[mjackson]

Yes, some help would be very much appreciated. I think implementation should be pretty straightforward.

Let's pretend we're writing some user-facing docs for this feature. How could we describe it in a sentence or two?

Here's a first draft:

You can use the 'as' query parameter to specify a directory that unpkg should look in if the file in the original request path is not found. This is useful when using SystemJS to include packages that keep their browser builds in different directories but do not use the browser field in their package.json.

Is that last part wrong? It seems like we're incurring extra complexity on our side because packages are not using the "browser" field properly, but there may still be a piece I'm missing.

[truongsinh]

As stated in https://github.com/unpkg/npm-http-server/issues/57, i'm happy to make MR

[mjackson]

Go for it! Can you also please include an example app that demonstrates the functionality you're after? I still haven't had time to look into this properly (sorry!) and I think a sample app might help me understand better.

[mjackson]

@truongsinh Have you had a second to look further into this? I'd really love to see a simple little app that I can just npm start to get the idea of what you guys are trying to do.

[mjackson]

@sebastien If you're still up for it, I'd love a hand with this. It sounds very promising.

[sebastien]

I'm not sure what to add to what I've already written. Basically, there are two new features/changes to implement:

1. Detect a dist/build version and serve it when passed the "as=umd" parameter. The behaviour should be: a) look for a browser entry in package.json, b) look for a dist or umd directory c) use it as the new root for serving files.

2. Make sure that any path ending with a slash shows the directory content, no slash can default to an index.js files as it is currently the case.

The end goal of this is to allow unpkg to be used as a source of directly usable JavaScript packages, without having to use any packager manager at all. The problem that these changes try to solve is the variety of configurations of where the pre-built files are located.

[mjackson]

Thanks for the summary, @sebastien. Would you care to make a PR? Otherwise I may have some time soon to take a look.

[sebastien]

I sadly don't have any experience with NodeJS server development :/

[alexisvincent]

checkout https://github.com/unpkg/npm-http-server/issues/65

[mjackson]

Just a note about this:

If an index.js is present in this directory, it should be served by default when the request URL path does not end with a /. For instance https://unpkg.com/tgd@0.0.4/umd/ and https://unpkg.com/tgd@0.0.4/umd now return the same, while it could https://unpkg.com/tgd@0.0.4/umd return the contents of https://unpkg.com/tgd@0.0.4/umd/index.js (or at least redirect to it).

https://unpkg.com/tgd@0.0.4/umd now redirects to https://unpkg.com/tgd@0.0.4/umd/index.js

I'm not sure why I didn't see this before. It seems obvious now that it always should've worked that way to preserve relative module paths.

Anyway, I think this should get us closer to being able to load stuff directly using SystemJS.

[billiegoose]

I just opened a case ModuleLoader/es-module-loader#538 that might be of interest. Basically I noticed that relative URLs are being resolved wrt the import URL rather than the import URL after redirects, and think that may be a bug in Chrome or an opportunity to improve the spec.

[abhishiv]

Hey guys

Have been trying to get it work for almost two years now.

Made many iterations, but always had some issue or another. But recently, I have been successful in using npm5's package-lock.json along with some simple post processing as source for systemjs config.

Post processing is necessary for two things

1. to make sure require('src/common') maps to require('src/common/index')
2. to deal with issues with some jspm-nodelibs-* packages - these have jspmNodeConversion = false in their package.json and I need to copy the map object from their package.json to make them work.

Code can be found here if anyone is interested abhishiv/systemjs-npm-resolver

demo here

[odahcam]

I'm a little late here, but I started to use SystemJS with a PHP env in past year and I want to know: can I use unpkg as a SystemJS source now or it stills kinda buggy?