lockfile support

[guybedford]

We currently have lockfile support in the installer via https://github.com/jspm/generator/blob/main/src/install/installer.ts#L60 along with the install option freeze, which when used will even freeze any updates of dependencies from that provided lockfile.

It would be great to expose this option to the Generator constructor so that users could pass a custom lockfile.

The lock format is currently typed at https://github.com/jspm/generator/blob/main/src/install/installer.ts#L36, and takes the following form:

{
  "file:///path/to/pkg/": {
    "dep": "https://ga.jspm.io/npm:pkg@version/"
  }
}

The main base URL used can just be generator.baseUrl for the local resolutions. It is effectively the URL of the package.json folder that defines the dependencies.

It is important that all URLs are fully normalized - thus the initial input into the lockfile system needs to deal with normalization into this format.

The standard library resolutions for Node.js use a special symbol | in the lockfile to indicate packages that are installed to export values.

For example, fs would be written:

{
  "file:///path/to/baseurl/": {
    "fs": "https://ga.jspm.io/npm:@jspm/core@2.1.0/|fs"
  }
}

Which ensures that it will always resolve to the correct environment fs library against the JSPM core package on the CDN. This is the only exception case to worry about.

To generate this lock format from an input map should be relatively straightforward based on effectively just iterating over the import map and collecting the resolutions.

We could then automatically support treating a file with "imports" or "scopes" as an import map input, versus the direct lockfile input. We could possibly even handle automatic support for npm and Yarn lockfile formats as further work in future but that isn't necessary for the MVP.

Steps:

1. Expose "lockfile" option, supporting this format exactly, just with an iteration that converts all URLs into absolute URLs relative to the baseUrl of the generator so that a proper normalized lockfile is passed in.
2. Support detecting "imports" and "scopes" on this object then converting an import map input into this same lockfile format through an iterative reduction. For a given URL https://path/to/module.js, to determine the package boundary, there is already a resolver.getPackageBase(url) function which can be reliably used for any URL in a way that works with the provider hooks properly.
3. Profit!

[jwfwessels]

Thanks for the thorough issue @guybedford,

Some questions for edification

• Would the above solution also work for an import map with a resolution that points to a module(single file) self-hosted, for example, since no pkg can be extracted from this as there is no authoritative package.json for metadata?

• Do you have a way to identify Node standard libraries? else how would I determine that they need the particular exports format with the | (I find that part of the requirement a bit confusing I will try to read up on it a bit more)

• Lastly, wouldn't it be easier to use an import map directly as the lock file? That is, use the resolutions as is? Why do we extract the packages and recalculate them? Is it for situations where you want multiple entry points from a package 🤔 .

[guybedford]

Would the above solution also work for an import map with a resolution that points to a module(single file) self-hosted, for example, since no pkg can be extracted from this as there is no authoritative package.json for metadata?

Yes, the getPackageBase(url) works for all urls, and the lockfile itself is URL-based instead of namespace-based for this reason. For URL modules without a package.json at any of the URL subpaths (these are checked via backtracking), the root origin URL is used as the package scope.

Do you have a way to identify Node standard libraries? else how would I determine that they need the particular exports format with the | (I find that part of the requirement a bit confusing I will try to read up on it a bit more)

The | format is not specific to Node standard libraries and can work with any package URL. But yes, reversing this format in the lockfile conversion is a very good point. Effectively it's applying a backwards trace on the "exports" field of the package - working out what was exported to achieve the original import map. For example JSPM core has this package.json - https://github.com/jspm/jspm-core/blob/main/package.json#L44. For https://ga.jspm.io/npm:@jspm/core@2.0.0-beta.x/nodelibs/browser/fs.js one could determine from the exports field that this was originally the nodelibs/fs export and thus that the package install for fs is @jspm/core@2.0.0-beta.x/|nodelibs/fs.

The above may seem like unnecessary complexity but it is very much necessary complexity, since the import map represents a single environment resolution, while the exports field permits multiple environment resolutions. The work in the above effectively regeneralizes the lock from the import map and was one of the big problems with lockfiles I've always been stuck on. In talking it through with you here, the whole problem just made sense to me in this context for the first time so thank for poking this!

Lastly, wouldn't it be easier to use an import map directly as the lock file? That is, use the resolutions as is? Why do we extract the packages and recalculate them? Is it for situations where you want multiple entry points from a package 🤔 .

Yes, the import map only works at the module level, for additional exports we need to know what is going on at the package level otherwise you could get different version resolutions for different entry points of the same package. It's that simple. And as per the above, the added benefit is that if you want to take an import map and regenerate it for Deno or Node.js or another platform instead of the browser, this work achieves that too.

[jwfwessels]

You are very welcome! Though I'm still wrapping my head around all of it 😅.

[guybedford]

Okay, let me know if I can help clarify at all further @jwfwessels, I'm also happy to pair on the problem as well if you like.

[guybedford]

@jwfwessels also if you would rather I make a start on this just let me know too and I'll add it to my queue, just don't want to duplicate work either.

[jwfwessels]

@guybedford I'm hoping to loop back to this later this week. I think once I have something going, a pairing session would be great!

[guybedford]

Added in #104.