Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to convert package-lock.json to import maps? #60

Open
wmhilton opened this issue Oct 9, 2018 · 20 comments

Comments

Projects
None yet
10 participants
@wmhilton
Copy link

commented Oct 9, 2018

This looks so cool! Are there any implementations of "package-lock.json -> package-name-map" translators yet? (Is that what I should be thinking of?)

(Edit by @domenic: at the time this comment was written the proposal was known as "package name maps". I've renamed the title to be "import maps", but left the body of subsequent comments alone. Sorry if that's confusing!)

@adrianhelvik

This comment has been minimized.

Copy link

commented Oct 15, 2018

Something like this should do it. You would use package.json and not package-lock.json.

const fs = require('fs')

const packageMap = fs
  .readdirSync(
    __dirname
    + '/node_modules'
  )
  .reduce((res, name) => {
    try {
      var package =  require(folder + '/package')
    } catch (e) {
      console.log(folder + ' had no package.json. Skipping...')
      return res
    }
    if (! package.module) {
      console.log(`"${name}" had no module filed. Skipping...`)
      return res
    }
    res.packages[name] = {
      main: package.module
    }
    return res
  }, {
    path_prefix: '/node_modules',
    packages: {}
  })

fs.writeFileSync(
  __dirname + '/package-map.json',
  JSON.stringify(packageMap, null, 2)
)
@tilgovi

This comment has been minimized.

Copy link

commented Oct 15, 2018

I would think you would want to package-lock.json unless you can guarantee that all dependencies are hoisted and totally flat.

@adrianhelvik

This comment has been minimized.

Copy link

commented Oct 15, 2018

Ah, I didn't consider transitive deps.

@adrianhelvik

This comment has been minimized.

Copy link

commented Oct 16, 2018

Came to think of it. Do packages support a packages field? This would be crucial for node interop.

Otherwise this will not be supported:

/node_modules
  /some-lib
    /node_modules
      /jquery (v2)
  /jquery (v3)
@wmhilton

This comment has been minimized.

Copy link
Author

commented Oct 16, 2018

Exactly, package maps cannot be generated directly from the "dependencies" field of package.json because they only specify the first-order dependencies and even those are only by a semver range.

(Edit: and crawling the filesystem is fraught with edge cases like symlinks. You'd have to use module.require.resolve for sure. It will quickly become ugly and complicated to try to generate a package-name-map from the filesystem. Which is why we should piggy-back on tools that take care of all that for us and parse the lockfiles they generate.)

The equivalent to a package-name-map is a "lockfile" format, which is usually created during an actual install.

I suggested package-lock.json because it's the format used by npm and therefore ought to be a defacto standard. (The npm CLI automatically produces a package-lock.json after nearly any operation.) However, for maximum developer happiness and we'd want to be able to convert:

  • package-lock.json
  • npm-shrinkwrap.json (same format, different use case)
  • yarn.lock (generated by yarn)
  • shrinkwrap.yaml (generated by pnpm)

So that's at least 3 CLI tools that need to be built, I take it?

@tilgovi

This comment has been minimized.

Copy link

commented Oct 16, 2018

The scopes feature allows what you're looking for, but see also #5.

@daKmoR

This comment has been minimized.

Copy link

commented Oct 16, 2018

also Yarn Plug'n'Play: Implementation should be considered - It comes with it's own file/api

More Info here: yarnpkg/yarn#6382

@wmhilton

This comment has been minimized.

Copy link
Author

commented Oct 20, 2018

Ah! very interesting. This pnp thing ships with a build-pnm (build package-name-map) tool as of last month. yarnpkg/pnp-sample-app@c36baa5

@daKmoR

This comment has been minimized.

Copy link

commented Oct 20, 2018

oh yeah, that looks promising... unfortunately not yet useful for web components.
I made a test with what is currently available

https://github.com/daKmoR/build-pnm-webcomponent-issue

I raised an issue there as well - hopefully, this will start a discussion about unique elements in a package name map. For anyone who is interested arcanis/build-pnm#1.

@arcanis

This comment has been minimized.

Copy link

commented Oct 20, 2018

Thanks for the ping! I'm entering a plane but I'll share some points regarding the build-pnm utility later today!

In the meantime, can you details the reasons why webcomponents are expected to be flat, with a single version used accross an application? Since the pnm standard seems to support nested packages I wonder if there is a technical requirement I've missed in the spec 😃

@daKmoR

This comment has been minimized.

Copy link

commented Oct 20, 2018

comment moved to: arcanis/build-pnm#1

@wmhilton

This comment has been minimized.

Copy link
Author

commented Oct 20, 2018

WebComponents is a significant change of topic from my original issue, which is asking what tools exist for creating package-name-maps. I would move that discussion to a new issue so as not to get them mixed up.

@domenic

This comment has been minimized.

Copy link
Collaborator

commented Nov 2, 2018

Looking forward to what people produce here. I'll note that the proposal has just changed radically (and been renamed), in ways that might make this more interesting. As such let me rename the issue.

@domenic domenic changed the title How to convert package-lock.json to package-name-map? How to convert package-lock.json to import maps? Nov 2, 2018

@CarterLi

This comment has been minimized.

Copy link

commented Apr 26, 2019

Generates import maps from yarn.lock

import * as lockfile from '@yarnpkg/lockfile';
import { promises as fs } from 'fs';
import * as path from 'path';

export async function getImportMap(targetPath = __dirname) {
  const content = await fs.readFile(path.resolve(targetPath, 'yarn.lock'), 'utf-8');
  const json = lockfile.parse(content);

  return Object.assign({}, ...Object.keys(json.object)
    .map(x => x.slice(0, x.lastIndexOf('@')))
    .map(x => {
      try {
        const result = '/' + path.relative(targetPath, require.resolve(x, { paths: [targetPath] }));
        return { [x]: result, [x + '/']: path.dirname(result) + '/' };
      } catch {
        return { [x]: undefined } ;
      }
    }));
}

A working demo that uses import maps in browser with node_modules: https://github.com/CarterLi/web-esm/blob/master/index.ts#L27

@ftaiolivista

This comment has been minimized.

Copy link

commented May 15, 2019

Generates import maps from yarn.lock

import * as lockfile from '@yarnpkg/lockfile';
import { promises as fs } from 'fs';
import * as path from 'path';

export async function getImportMap(targetPath = __dirname) {
  const content = await fs.readFile(path.resolve(targetPath, 'yarn.lock'), 'utf-8');
  const json = lockfile.parse(content);

  return Object.assign({}, ...Object.keys(json.object)
    .map(x => x.slice(0, x.lastIndexOf('@')))
    .map(x => {
      try {
        const result = '/' + path.relative(targetPath, require.resolve(x, { paths: [targetPath] }));
        return { [x]: result, [x + '/']: path.dirname(result) + '/' };
      } catch {
        return { [x]: undefined } ;
      }
    }));
}

A working demo that uses import maps in browser with node_modules: https://github.com/CarterLi/web-esm/blob/master/index.ts#L27

Would be great to filter out development packages and generate map only for production ones.

@CarterLi

This comment has been minimized.

Copy link

commented May 15, 2019

Would be great to filter out development packages and generate map only for production ones.

It's easy to filter out development packages themselves, but it's hard to filter out packages that dev packages depend on.

I dont know if yarn has an API to do so, or you will have to parse package.json in node_modules recursively

@ljharb

This comment has been minimized.

Copy link

commented May 15, 2019

package-lock already contains this information; anything with dev: true is something that's a dev dep, or a transitive dep of a dev dep that's not a transitive dep of a production dep (iow, safe to prune)

@dgreene1

This comment has been minimized.

Copy link

commented May 17, 2019

Would anyone be interested in turning the two snippets above into a CLI library so that we can have all the benefits of an open source package?

(I.e. modularization, the package can be upgraded centrally, etc)

@adrianhelvik

This comment has been minimized.

Copy link

commented May 17, 2019

@dgreene1 Agree that we need that, but it can't really use require.resolve. Now it works relative to the file and not cwd.

@ljharb

This comment has been minimized.

Copy link

commented May 17, 2019

the resolve package allows a custom basedir; you could use that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.