Skip to content
This repository has been archived by the owner on Aug 11, 2022. It is now read-only.

spec: Describe npm-shrinkwrap.json and package-lock.json #16441

Closed
wants to merge 2 commits into from

Conversation

iarna
Copy link
Contributor

@iarna iarna commented Apr 25, 2017

Here is my proposal for how shrinkwraps in npm@5 will work. We will also be introducing a new file, package-lock.json that will fill the same role as npm-shrinkwrap.json in projects that don't otherwise have a shrinkwrap.

The TLDR is:

You'll use an npm-shrinkwrap.json if you want to publish it so that users of your module will consume it.
Otherwise you'll use a package-lock.json. package-lock.json files will never be published but are suitable for checking into git.

This is open for feedback for the next two weeks. (I mean, we'll take feedback whenever, but it won't be immediately actionable beyond that.)

`package-lock.json` and `npm-shrinkwrap.json` are JSON documents described
below. The difference between a `package-lock.json` and an
`npm-shrinkwrap.json` is that the former is never published. The `npm shrinkwrap` command
renames the `package-lock.json` (or if missing, generates one.)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renames to npm-shrinkwrap.json?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jesstelford correct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'll clarify that!

* For git sources this is the specific commit hash we cloned from.
* For remote tarball sources this is an integrity based on a SHA512 of
the file.
* For local tarball sources: This is an integrity field based on the SHA512 of the file.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting that there will be potentially 3 different formats for integrity (SHA1, SHA512, the subresource Integrity format) - would this potentially increase complexity in consuming applications (such as npm, and maybe in the future yarn)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jessmartin there is exactly one format: subresource integrity. That standard supports any algorithm attached to tarballs. Integrity fields can also support multiple different hashes for the same data (with different algorithms). So, if you have SRI support, your tool will automatically pick up sha512 from registries that serve those.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, I hadn't thought through those extra use-cases. Thanks for the explanation 👍

@felixrieseberg
Copy link
Contributor

felixrieseberg commented Apr 25, 2017

Hey Rebecca, fantastic effort 👏 I'm so happy you're putting some focus on shrinkwraps, they're core to releases we make. Thank you!

One thing I'd like to call out: Cross-platform shrinkwraps. You cover that with "All optional dependencies should be included even if they're uninstallable on the current platform", but I'd love to see this be a reliable feature of shrinkwraps.

npm is possibly the best cross-platform installation tool available right now. At Slack and in other projects, I use shrinkwraps to describe the exact dependency tree of a cross-platform project at compilation time. We currently merge shrinkwraps generated on each platform by hand, but we'd love to be able to generate the "one true master shrinkwrap" automatically.

@zkat
Copy link
Contributor

zkat commented Apr 25, 2017

@felixrieseberg This spec also covers the shrinkwrap behavior. The difference between package-lock.json and npm-shrinkwrap.json is that the latter is meant for publication, and the former is meant for devs to commit. The rest is the same (modulo the fact that this new version of the lockfile has different fields -- older npms will still be able to install with shrinkwraps generated by this though)

@vinkla
Copy link

vinkla commented Apr 25, 2017

Great idea 🎉 I think this will make it easier for newcomers to understand what the lock file is.

We will also be introducing a new file, package-lock.json that will fill the same role as npm-shrinkwrap.json in projects that don't otherwise have a shrinkwrap.

What about naming the lock file to package.lock similar to how Composer and Yarn handles lock files? In my opinion it would make it even easier to understand.

package.json
package.lock

Reference in the Composer documentation: https://getcomposer.org/doc/01-basic-usage.md#installing-with-composer-lock

@iamstarkov
Copy link

i know @zkochan worked hard to get reliable lock file for pnpm. he might has some input

@iamstarkov
Copy link

@vinkla package.lock seems like a nice filename

@zkat
Copy link
Contributor

zkat commented Apr 25, 2017

@vinkla The current name is clearer about the file format. require('./package-lock.json') will give you a parsed JSON object with this. Other tools that look at the extension will work unaltered (for example, webpack, editors, etc).

(I don't really see much of a difference between package.lock and package-lock.json as far as clarity goes. It seems to me like it would be pretty minor, and unlikely to be significant enough to sacrifice the occasional JSON tooling convenience.)

@zkat zkat mentioned this pull request Apr 25, 2017
24 tasks
@vinkla
Copy link

vinkla commented Apr 25, 2017

@zkat I understand the file format issue and I agree .json makes it clearer what kind of content the file contains. Though, in my opinion it is clearer that the file ending with lock locks the dependencies in the package.json file. It also makes the file tree look clean.

package.json
package.lock

For newcomers, I think package.json and package-lock.json will take longer to connect than what package.json and package.lock would.

It would be cool if we were to sync the naming convention with Yarn and Composer. Then developers getting started with npm coming from another package manager such as Yarn or Composer would be familiar with the lock file extension. This would also work the other way around.

### `npm shrinkwrap`

If a `package-lock.json` exists, rename it to `npm-shrinkwrap.json`.
Refresh the data from the installer's ideal tree.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the "ideal tree" and the "currently installed tree" differ? Could this lead to a different dependency tree for a co-worker / on deployment / etc?

Edit:

As you are no longer going to be allowed to put your node_modules in a state that's not a valid shrinkwrap [...]

I'm not 100% familiar with the changes in npm@5; does this mean my question above is invalid? Ie; is it never possible to get the currently installed tree and the "ideal tree" out of sync (when using npm commands)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jesstelford the two main cases where these trees might deviate are:

  • npm i --only=prod - this removes dev: true deps
  • optional dependencies (when they fail to build on certain platforms)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, if you have optional dependencies that couldn't be installed, or you're running with --production then your currently installed tree wouldn't have those, but the ideal tree would

`integrity`.
* Otherwise, try these in turn and validate the result against the `integrity`:
* `resolved`, then `from`, then `version.
* `from` can be either `package@specifier` or just `specifier`.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this downgrade path produce a warning?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could, but also this would mean you get warnings for published npm packages that publish an npm-shrinkwrap.json. I don't feel like it has that much value if it will be adapted next time someone runs a new install.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting - a nested shrinkwrap is honoured? I thought the top level shrinkwrap would include all the nested dependencies already, and as such would be generated with the tooling being run by the developer (not the 3rd party module author)?

The specific case I was thinking about is when deploying to production involves an npm i: The downgrade could potentially mean MITM, brining in a semver minor when you wanted to lock it to semver patch ranges only, etc. All of which could introduce either breaking changes, or worse; hidden security holes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @iarna but I'm pretty sure the toplevel lockfile overrides literally everything else. Nested shrinkwraps are only used during installation of that child package.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right. The deeper shrinkwrap would be consulted when first installing the module (when adding it to your project) but it won't be consulted ever again as all of its information will end up in your top level shrinkwrap.

@iarna
Copy link
Contributor Author

iarna commented Apr 25, 2017

@felixrieseberg Regarding cross platformness, I'll make sure it's clearer, but yes, ALL optional dependencies, ALL dev dependencies should be in lock/shrinkwrap files even if you installed with --no-optional and --only=production.

Copy link

@jesstelford jesstelford left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent stuff - looking forward to it!

@zkochan
Copy link

zkochan commented Apr 25, 2017

What about some of the configurations found in npmrc?

Shouldn't they be included in the lock file? I mean things like: no-bin-links, ignore-scripts. Probably there are more. Configs that can change the way dependencies are installed

@iarna
Copy link
Contributor Author

iarna commented Apr 26, 2017

@zkochan No, I believe those belong in a project-local .npmrc. There is one that that I think does belong in a lock/shrinkwrap and that's node's preserve-symlinks (env: NODE_PRESERVE_SYMLINKS).

@novemberborn
Copy link

How would one update a package-lock.json if new dependencies were added by manually editing package.json? Presumably running npm install would install those dependencies, but it won't update package-lock.json.


The version of the package this is a shrinkwrap for. This must match what's in `package.json`.

### created-with *(new)*
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be better to use the same camelCase conventions for naming properties which are used in package.json?

* For bundled dependencies this is not included, regardless of source.
* For registry sources this is path of the tarball relative to the registry
URL. If the tarball URL isn't on the same server as the registry URL then
this is a complete URL.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the tarball URL is not the same server as the registry URL, where can I find the actual registry URL from the lockfile?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zkochan the intention is to be able to switch registries and have it continue to work. So, npm config get registry :)

This is something the CLI's already been doing since forever and a day by munging tarball URLs internally, but we thought it might be nicer to make the "relative-ness" of these tarball URLs more explicit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zkat By switch registries, what do you mean?

Copy link

@Daniel15 Daniel15 May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@legodude17 - I think "switch registries" means switching from the main upstream registry (registry.npmjs.org) to either a mirror or a completely different server that's a subset of the npmjs registry. For example, CloudFlare's mirror at https://www.npmjs.cf/, Taobao's mirror for Chinese users at https://npm.taobao.org/, or an internal mirror for corporate environments.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A similar issue is documented here - yarnpkg/yarn#2566

Being able to switch registries for packages specified in the lockfile will be really helpful.

This is a record of what specifier was used to originally install this
package. This should not be included in new `npm-shrinkwrap.json` files.

#### dependencies
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason to have this nesting?

For pnpm we have a flat mapping, <pkgId>: <resolution>. Like this:

  /ansi-regex/2.1.1: c3b33ab5ee360d86e0e628f0468ae7ef27d654df
  /ansi-styles/2.2.1: b432dd3358b634cf75e1e4664368240533c1ddbe
  /chalk/1.1.3:
    dependencies:
      ansi-styles: 2.2.1
      escape-string-regexp: 1.0.5
      has-ansi: 2.0.0
      strip-ansi: 3.0.1
      supports-color: 2.0.0
    resolution: a8115c55e4a702fe4d150abd3872822a7e09fc98

We create one copy of the lockfile inside node_modules and one outside. Comparing the two lockfiles can very easily give us valuable info about the state of the dependency tree.

We can just compare the two lockfiles and say if we need to run npm install. We can look what packages are missing from the inner lockfile and tell what can be pruned.

Maybe with this structure it is possible to do these manipulations as well. But IMHO, with a flat key/value structure algorithms become simpler.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The layout describes the structure of the node_modules direcotry to be created and insures that you'll get the same result every time. Without this you're at the mercy of the algorithm of the installer. If that changes, you suddenly have a different layout. While modules shouldn't depend on particular layouts, we know from hard experience that they do.

@iarna
Copy link
Contributor Author

iarna commented Apr 26, 2017

@novemberborn You'll note that the package-lock has a integrity hash generated from the package.json. Ah, I didn't mention that when describing install semantics, I'll correct that.

Copy link
Contributor

@legodude17 legodude17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fast installs for everyone!

## Managing the two files

If at any time both `package-lock.json` and an `npm-shrinkwrap.json` exist
then the former should be unlinked with a warning the latter used. This should happen on any command
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest putting an and there:

...with a warning and the latter used...

`package-lock.json` and `npm-shrinkwrap.json` are JSON documents described
below. The difference between a `package-lock.json` and an
`npm-shrinkwrap.json` is that the former is never published. The `npm shrinkwrap` command
renames the `package-lock.json` (or if missing, generates one.)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it generate a package-lock.json or an npm-shrinkwrap.json?


### package-integrity *(new)*

A hash of the `package.json` that was in use when this shrinkwrap was
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What algorithm?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

* For bundled dependencies this is not included, regardless of source.
* For registry sources this is path of the tarball relative to the registry
URL. If the tarball URL isn't on the same server as the registry URL then
this is a complete URL.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zkat By switch registries, what do you mean?

If true then this dependency is either a development dependency ONLY of the
top level module or a transitive dependency of one. This is false for
dependencies that are both a development dependency of the top level and a
transitive dependency of a non-development dependency of the top level.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about if it is a dev dep and a dep of the top project?

All optional dependencies should be included even if they're uninstallable
on the current platform.

#### from *(deprecated)*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is for backwards compat?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It just means that you're likely to run into it because of older shrinkwraps (which can be processed by this spec!)

So. Don't add a field called from.


#### dependencies

The dependencies of this dependency, exactly as at the top level.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't this lead to an infinite loop? Like, top is A. A depends on B which depends on C which depends on B. The installer could handle this by flattening the tree, but what about the shrinkwrap? Does it map out the flattened tree?

### `npm install --save`

If either an `npm-shrinkwrap.json` or a `package-lock.json` exists then it
will be updated.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about if both exist?

* Install using the value of `version` and validate the result against the
`integrity`.
* Otherwise, try these in turn and validate the result against the `integrity`:
* `resolved`, then `from`, then `version.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you missed the closing backtick on version..

that users would expect `npm rm pkgname` to be equivalent of
`rm -rf node_modules/pkgname`.

As you are no longer going to be allowed to put your `node_modules` in a
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would npm install or npm shrinkwrap do if you manually edit your node_modules to put it in an invalid state?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is up to the individual installers, I think. npm reads and verifies and fixes your trees. Others rely exclusively on a hash in a dotfile file in node_modules and trust the user. I think it's ok to assume this is out of scope for this RFC

@novemberborn
Copy link

@iarna

You'll note that the package-lock has a integrity hash generated from the package.json. Ah, I didn't mention that when describing install semantics, I'll correct that.

Sure. Once it's out of sync though, how would I update the package-lock.json file?

@indexzero
Copy link
Contributor

🥇 well articulated & sensible. Really happy to see npm moving towards supporting lockfiles, but also introducing the semantics that yarn lacks. That is differentiating between publishable lockfiles (in this case npm-shrinkwrap.json) and non-publishable lockfiles for performance optimization of the average case (package-lock.json).

Copy link
Contributor

@zkat zkat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments around git and bundle deps stuff


* For bundled dependencies this is not included, regardless of source.
* For registry sources, this is the `integrity` that the registry provided, or if one wasn't provided the SHA1 in `shasum`.
* For git sources this is the specific commit hash we cloned from.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should put the sha512 of the generated tarball here anyway.

It might not be of much use, and the installer can understand enough about this to skip it for git deps if they don't match, but -- this is still useful if folks have already built a tarball locally, cause they'd still be able to do content-addressed lookups this way.

Once we have a tarball packer that can do sequential packs on the same data that have the same checksum, we'll be able to use this more often. We can just avoid verifying the generated tarballs with this, due to those differences.

Furthermore, the hash we cloned from is already in resolved. There's no need for this duplication.


If true, this is the bundled dependency and will be installed by the parent
module. When installing, this module will be extracted from the parent
module during the extract phase, not installed as a separate dependency.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be a little more specific here and say it will be extracted from the root bundler (the first package in the parent chain that doesn't have bundled on it). Bundles can be nested.

## Additional fields / Adding new fields

Installers should ignore any field they aren't aware of. It's not an error
to have additional properities in the shrinkwrap or lock file.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit: Typo "properities" → "properties"


Installers that want to add new fields should either have one added via RFC
in the npm issue tracker and an accompanying documentation PR, or should prefix
it with the name of their project.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should specify the format of the prefix - For example, it could be underscore separated ({project}_{fieldName}) or colon separated ({project}:{fieldName}) or something else altogether.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would you suggest?

CSS folks have a pretty "standard" familiar syntax: -brwsr-my-property:

I do like : or :: as separators though. :)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, I don't think it matters exactly what the format is, as long as it's documented. Just go with whatever you think looks best 😄 I do think : looks like a nice separator though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome! 🎉

zkat added a commit that referenced this pull request May 2, 2017
This implements #16441 as well as the auto-save behavior it needs in order to be kept up to date.
@alloy
Copy link

alloy commented May 2, 2017

Has there been any thought given into making the lockfile shareable with yarn?

@zkochan
Copy link

zkochan commented May 30, 2017

I think it would be nice to be able to detect which dependencies cannot be installed on the current system using just the lockfile. Currently, the package.json has to be downloaded in order to check the engines field. Having the engines field in the lockfiles would save us a few redundant HTTP requests.

What do you think?

It may be valuable info for code reviews as well

Mithgol added a commit to Mithgol/node-gamayun that referenced this pull request May 30, 2017
Mithgol added a commit to Mithgol/node-large-split that referenced this pull request May 30, 2017
Mithgol added a commit to Mithgol/node-truncate-escaped-html that referenced this pull request May 30, 2017
Mithgol added a commit to Mithgol/node-twi2fido that referenced this pull request May 30, 2017
@zkat
Copy link
Contributor

zkat commented May 30, 2017

@zkochan part of the issue is that the definition of "failed optional dep" is literally "tried to install it and it failed". The engines field has always been an FYI and we can't really treat it as normative, lest a lot of other stuff break, I think. /cc @iarna cause I'm pretty sure that's right.

Mithgol added a commit to Mithgol/node-unsquish that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/node-uue that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/node-uue2ipfs that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/node-webbbs that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/npmtree that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/phido that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/rss2fido that referenced this pull request Jun 1, 2017
Mithgol added a commit to Mithgol/rules-fe2hpt that referenced this pull request Jun 1, 2017
Mithgol added a commit to Mithgol/shee2arr that referenced this pull request Jun 1, 2017
@zkochan
Copy link

zkochan commented Jun 2, 2017

@zkat unless engine-strict is set to true

If set to true, then npm will stubbornly refuse to install (or even consider installing) any package that claims to not be compatible with the current Node.js version.

@isaacs
Copy link
Contributor

isaacs commented Jun 2, 2017

It's well after the time for comments on this particular aspect of the bikeshed, and what I wanted already happened, but .lock as a file extension tells me it's a file system mutex lock. I know yarn and gem don't use it that way, but I've interacted with many programs over the years that do. (For example, this is what a .lock file in an npm <4 cache is.)

The .json extension says "this contains meaningful data".

@gauravmuk
Copy link

gauravmuk commented Jun 15, 2017

Is there any way that I can not generate package-lock.json every time on npm install with npm@5?

EDIT: got the answer, --no-package-lock


This is a
[subresource integrity](https://w3c.github.io/webappsec/specs/subresourceintegrity/)
value created from the `pacakge.json`. No preprocessing of the

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: pacakge --> package :)

@zkat
Copy link
Contributor

zkat commented Jun 29, 2017

This has been merged 👍

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet