Skip to content
This repository has been archived by the owner. It is now read-only.

spec: Describe npm-shrinkwrap.json and package-lock.json #16441

Closed
wants to merge 2 commits into from

Conversation

@iarna
Copy link
Contributor

@iarna iarna commented Apr 25, 2017

Here is my proposal for how shrinkwraps in npm@5 will work. We will also be introducing a new file, package-lock.json that will fill the same role as npm-shrinkwrap.json in projects that don't otherwise have a shrinkwrap.

The TLDR is:

You'll use an npm-shrinkwrap.json if you want to publish it so that users of your module will consume it.
Otherwise you'll use a package-lock.json. package-lock.json files will never be published but are suitable for checking into git.

This is open for feedback for the next two weeks. (I mean, we'll take feedback whenever, but it won't be immediately actionable beyond that.)

@iarna iarna added the in-progress label Apr 25, 2017
`package-lock.json` and `npm-shrinkwrap.json` are JSON documents described
below. The difference between a `package-lock.json` and an
`npm-shrinkwrap.json` is that the former is never published. The `npm shrinkwrap` command
renames the `package-lock.json` (or if missing, generates one.)

This comment has been minimized.

@jesstelford

jesstelford Apr 25, 2017

Renames to npm-shrinkwrap.json?

This comment has been minimized.

@zkat

zkat Apr 25, 2017
Contributor

@jesstelford correct.

This comment has been minimized.

@iarna

iarna Apr 25, 2017
Author Contributor

Yes, I'll clarify that!

* For git sources this is the specific commit hash we cloned from.
* For remote tarball sources this is an integrity based on a SHA512 of
the file.
* For local tarball sources: This is an integrity field based on the SHA512 of the file.

This comment has been minimized.

@jesstelford

jesstelford Apr 25, 2017

Interesting that there will be potentially 3 different formats for integrity (SHA1, SHA512, the subresource Integrity format) - would this potentially increase complexity in consuming applications (such as npm, and maybe in the future yarn)?

This comment has been minimized.

@zkat

zkat Apr 25, 2017
Contributor

@jessmartin there is exactly one format: subresource integrity. That standard supports any algorithm attached to tarballs. Integrity fields can also support multiple different hashes for the same data (with different algorithms). So, if you have SRI support, your tool will automatically pick up sha512 from registries that serve those.

This comment has been minimized.

@jesstelford

jesstelford Apr 25, 2017

Interesting, I hadn't thought through those extra use-cases. Thanks for the explanation 👍

@felixrieseberg
Copy link
Contributor

@felixrieseberg felixrieseberg commented Apr 25, 2017

Hey Rebecca, fantastic effort 👏 I'm so happy you're putting some focus on shrinkwraps, they're core to releases we make. Thank you!

One thing I'd like to call out: Cross-platform shrinkwraps. You cover that with "All optional dependencies should be included even if they're uninstallable on the current platform", but I'd love to see this be a reliable feature of shrinkwraps.

npm is possibly the best cross-platform installation tool available right now. At Slack and in other projects, I use shrinkwraps to describe the exact dependency tree of a cross-platform project at compilation time. We currently merge shrinkwraps generated on each platform by hand, but we'd love to be able to generate the "one true master shrinkwrap" automatically.

@zkat
Copy link
Contributor

@zkat zkat commented Apr 25, 2017

@felixrieseberg This spec also covers the shrinkwrap behavior. The difference between package-lock.json and npm-shrinkwrap.json is that the latter is meant for publication, and the former is meant for devs to commit. The rest is the same (modulo the fact that this new version of the lockfile has different fields -- older npms will still be able to install with shrinkwraps generated by this though)

@vinkla
Copy link

@vinkla vinkla commented Apr 25, 2017

Great idea 🎉 I think this will make it easier for newcomers to understand what the lock file is.

We will also be introducing a new file, package-lock.json that will fill the same role as npm-shrinkwrap.json in projects that don't otherwise have a shrinkwrap.

What about naming the lock file to package.lock similar to how Composer and Yarn handles lock files? In my opinion it would make it even easier to understand.

package.json
package.lock

Reference in the Composer documentation: https://getcomposer.org/doc/01-basic-usage.md#installing-with-composer-lock

@iamstarkov
Copy link

@iamstarkov iamstarkov commented Apr 25, 2017

i know @zkochan worked hard to get reliable lock file for pnpm. he might has some input

@iamstarkov
Copy link

@iamstarkov iamstarkov commented Apr 25, 2017

@vinkla package.lock seems like a nice filename

@zkat
Copy link
Contributor

@zkat zkat commented Apr 25, 2017

@vinkla The current name is clearer about the file format. require('./package-lock.json') will give you a parsed JSON object with this. Other tools that look at the extension will work unaltered (for example, webpack, editors, etc).

(I don't really see much of a difference between package.lock and package-lock.json as far as clarity goes. It seems to me like it would be pretty minor, and unlikely to be significant enough to sacrifice the occasional JSON tooling convenience.)

@zkat zkat mentioned this pull request Apr 25, 2017
16 of 24 tasks complete
@vinkla
Copy link

@vinkla vinkla commented Apr 25, 2017

@zkat I understand the file format issue and I agree .json makes it clearer what kind of content the file contains. Though, in my opinion it is clearer that the file ending with lock locks the dependencies in the package.json file. It also makes the file tree look clean.

package.json
package.lock

For newcomers, I think package.json and package-lock.json will take longer to connect than what package.json and package.lock would.

It would be cool if we were to sync the naming convention with Yarn and Composer. Then developers getting started with npm coming from another package manager such as Yarn or Composer would be familiar with the lock file extension. This would also work the other way around.

### `npm shrinkwrap`

If a `package-lock.json` exists, rename it to `npm-shrinkwrap.json`.
Refresh the data from the installer's ideal tree.

This comment has been minimized.

@jesstelford

jesstelford Apr 25, 2017

Could the "ideal tree" and the "currently installed tree" differ? Could this lead to a different dependency tree for a co-worker / on deployment / etc?

Edit:

As you are no longer going to be allowed to put your node_modules in a state that's not a valid shrinkwrap [...]

I'm not 100% familiar with the changes in npm@5; does this mean my question above is invalid? Ie; is it never possible to get the currently installed tree and the "ideal tree" out of sync (when using npm commands)?

This comment has been minimized.

@zkat

zkat Apr 25, 2017
Contributor

@jesstelford the two main cases where these trees might deviate are:

  • npm i --only=prod - this removes dev: true deps
  • optional dependencies (when they fail to build on certain platforms)

This comment has been minimized.

@iarna

iarna Apr 25, 2017
Author Contributor

Sure, if you have optional dependencies that couldn't be installed, or you're running with --production then your currently installed tree wouldn't have those, but the ideal tree would

`integrity`.
* Otherwise, try these in turn and validate the result against the `integrity`:
* `resolved`, then `from`, then `version.
* `from` can be either `package@specifier` or just `specifier`.

This comment has been minimized.

@jesstelford

jesstelford Apr 25, 2017

Should this downgrade path produce a warning?

This comment has been minimized.

@zkat

zkat Apr 25, 2017
Contributor

It could, but also this would mean you get warnings for published npm packages that publish an npm-shrinkwrap.json. I don't feel like it has that much value if it will be adapted next time someone runs a new install.

This comment has been minimized.

@jesstelford

jesstelford Apr 25, 2017

Interesting - a nested shrinkwrap is honoured? I thought the top level shrinkwrap would include all the nested dependencies already, and as such would be generated with the tooling being run by the developer (not the 3rd party module author)?

The specific case I was thinking about is when deploying to production involves an npm i: The downgrade could potentially mean MITM, brining in a semver minor when you wanted to lock it to semver patch ranges only, etc. All of which could introduce either breaking changes, or worse; hidden security holes.

This comment has been minimized.

@zkat

zkat Apr 25, 2017
Contributor

cc @iarna but I'm pretty sure the toplevel lockfile overrides literally everything else. Nested shrinkwraps are only used during installation of that child package.

This comment has been minimized.

@iarna

iarna Apr 26, 2017
Author Contributor

That's right. The deeper shrinkwrap would be consulted when first installing the module (when adding it to your project) but it won't be consulted ever again as all of its information will end up in your top level shrinkwrap.

@iarna
Copy link
Contributor Author

@iarna iarna commented Apr 25, 2017

@felixrieseberg Regarding cross platformness, I'll make sure it's clearer, but yes, ALL optional dependencies, ALL dev dependencies should be in lock/shrinkwrap files even if you installed with --no-optional and --only=production.

Copy link

@jesstelford jesstelford left a comment

Excellent stuff - looking forward to it!

@zkochan
Copy link

@zkochan zkochan commented Apr 25, 2017

What about some of the configurations found in npmrc?

Shouldn't they be included in the lock file? I mean things like: no-bin-links, ignore-scripts. Probably there are more. Configs that can change the way dependencies are installed

@iarna
Copy link
Contributor Author

@iarna iarna commented Apr 26, 2017

@zkochan No, I believe those belong in a project-local .npmrc. There is one that that I think does belong in a lock/shrinkwrap and that's node's preserve-symlinks (env: NODE_PRESERVE_SYMLINKS).

@novemberborn
Copy link

@novemberborn novemberborn commented Apr 26, 2017

How would one update a package-lock.json if new dependencies were added by manually editing package.json? Presumably running npm install would install those dependencies, but it won't update package-lock.json.


The version of the package this is a shrinkwrap for. This must match what's in `package.json`.

### created-with *(new)*

This comment has been minimized.

@zkochan

zkochan Apr 26, 2017

Wouldn't it be better to use the same camelCase conventions for naming properties which are used in package.json?

* For bundled dependencies this is not included, regardless of source.
* For registry sources this is path of the tarball relative to the registry
URL. If the tarball URL isn't on the same server as the registry URL then
this is a complete URL.

This comment has been minimized.

@zkochan

zkochan Apr 26, 2017

If the tarball URL is not the same server as the registry URL, where can I find the actual registry URL from the lockfile?

This comment has been minimized.

@zkat

zkat Apr 26, 2017
Contributor

@zkochan the intention is to be able to switch registries and have it continue to work. So, npm config get registry :)

This is something the CLI's already been doing since forever and a day by munging tarball URLs internally, but we thought it might be nicer to make the "relative-ness" of these tarball URLs more explicit.

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

@zkat By switch registries, what do you mean?

This comment has been minimized.

@Daniel15

Daniel15 May 2, 2017

@legodude17 - I think "switch registries" means switching from the main upstream registry (registry.npmjs.org) to either a mirror or a completely different server that's a subset of the npmjs registry. For example, CloudFlare's mirror at https://www.npmjs.cf/, Taobao's mirror for Chinese users at https://npm.taobao.org/, or an internal mirror for corporate environments.

This comment has been minimized.

@ghost

ghost May 3, 2017

A similar issue is documented here - yarnpkg/yarn#2566

Being able to switch registries for packages specified in the lockfile will be really helpful.

This is a record of what specifier was used to originally install this
package. This should not be included in new `npm-shrinkwrap.json` files.

#### dependencies

This comment has been minimized.

@zkochan

zkochan Apr 26, 2017

Is there any reason to have this nesting?

For pnpm we have a flat mapping, <pkgId>: <resolution>. Like this:

  /ansi-regex/2.1.1: c3b33ab5ee360d86e0e628f0468ae7ef27d654df
  /ansi-styles/2.2.1: b432dd3358b634cf75e1e4664368240533c1ddbe
  /chalk/1.1.3:
    dependencies:
      ansi-styles: 2.2.1
      escape-string-regexp: 1.0.5
      has-ansi: 2.0.0
      strip-ansi: 3.0.1
      supports-color: 2.0.0
    resolution: a8115c55e4a702fe4d150abd3872822a7e09fc98

We create one copy of the lockfile inside node_modules and one outside. Comparing the two lockfiles can very easily give us valuable info about the state of the dependency tree.

We can just compare the two lockfiles and say if we need to run npm install. We can look what packages are missing from the inner lockfile and tell what can be pruned.

Maybe with this structure it is possible to do these manipulations as well. But IMHO, with a flat key/value structure algorithms become simpler.

This comment has been minimized.

@iarna

iarna Apr 26, 2017
Author Contributor

The layout describes the structure of the node_modules direcotry to be created and insures that you'll get the same result every time. Without this you're at the mercy of the algorithm of the installer. If that changes, you suddenly have a different layout. While modules shouldn't depend on particular layouts, we know from hard experience that they do.

@iarna
Copy link
Contributor Author

@iarna iarna commented Apr 26, 2017

@novemberborn You'll note that the package-lock has a integrity hash generated from the package.json. Ah, I didn't mention that when describing install semantics, I'll correct that.

Copy link
Contributor

@legodude17 legodude17 left a comment

Fast installs for everyone!

## Managing the two files

If at any time both `package-lock.json` and an `npm-shrinkwrap.json` exist
then the former should be unlinked with a warning the latter used. This should happen on any command

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

I would suggest putting an and there:

...with a warning and the latter used...

`package-lock.json` and `npm-shrinkwrap.json` are JSON documents described
below. The difference between a `package-lock.json` and an
`npm-shrinkwrap.json` is that the former is never published. The `npm shrinkwrap` command
renames the `package-lock.json` (or if missing, generates one.)

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

Does it generate a package-lock.json or an npm-shrinkwrap.json?


### package-integrity *(new)*

A hash of the `package.json` that was in use when this shrinkwrap was

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

What algorithm?

This comment has been minimized.

@zkat

zkat Apr 26, 2017
Contributor

* For bundled dependencies this is not included, regardless of source.
* For registry sources this is path of the tarball relative to the registry
URL. If the tarball URL isn't on the same server as the registry URL then
this is a complete URL.

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

@zkat By switch registries, what do you mean?

If true then this dependency is either a development dependency ONLY of the
top level module or a transitive dependency of one. This is false for
dependencies that are both a development dependency of the top level and a
transitive dependency of a non-development dependency of the top level.

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

What about if it is a dev dep and a dep of the top project?

All optional dependencies should be included even if they're uninstallable
on the current platform.

#### from *(deprecated)*

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

So this is for backwards compat?

This comment has been minimized.

@zkat

zkat Apr 26, 2017
Contributor

It just means that you're likely to run into it because of older shrinkwraps (which can be processed by this spec!)

So. Don't add a field called from.


#### dependencies

The dependencies of this dependency, exactly as at the top level.

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

Couldn't this lead to an infinite loop? Like, top is A. A depends on B which depends on C which depends on B. The installer could handle this by flattening the tree, but what about the shrinkwrap? Does it map out the flattened tree?

### `npm install --save`

If either an `npm-shrinkwrap.json` or a `package-lock.json` exists then it
will be updated.

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

What about if both exist?

* Install using the value of `version` and validate the result against the
`integrity`.
* Otherwise, try these in turn and validate the result against the `integrity`:
* `resolved`, then `from`, then `version.

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

I think you missed the closing backtick on version..

that users would expect `npm rm pkgname` to be equivalent of
`rm -rf node_modules/pkgname`.

As you are no longer going to be allowed to put your `node_modules` in a

This comment has been minimized.

@legodude17

legodude17 Apr 26, 2017
Contributor

What would npm install or npm shrinkwrap do if you manually edit your node_modules to put it in an invalid state?

This comment has been minimized.

@zkat

zkat Apr 26, 2017
Contributor

This one is up to the individual installers, I think. npm reads and verifies and fixes your trees. Others rely exclusively on a hash in a dotfile file in node_modules and trust the user. I think it's ok to assume this is out of scope for this RFC

@novemberborn
Copy link

@novemberborn novemberborn commented Apr 27, 2017

@iarna

You'll note that the package-lock has a integrity hash generated from the package.json. Ah, I didn't mention that when describing install semantics, I'll correct that.

Sure. Once it's out of sync though, how would I update the package-lock.json file?

@indexzero
Copy link
Contributor

@indexzero indexzero commented Apr 27, 2017

🥇 well articulated & sensible. Really happy to see npm moving towards supporting lockfiles, but also introducing the semantics that yarn lacks. That is differentiating between publishable lockfiles (in this case npm-shrinkwrap.json) and non-publishable lockfiles for performance optimization of the average case (package-lock.json).

Copy link
Contributor

@zkat zkat left a comment

Added some comments around git and bundle deps stuff


* For bundled dependencies this is not included, regardless of source.
* For registry sources, this is the `integrity` that the registry provided, or if one wasn't provided the SHA1 in `shasum`.
* For git sources this is the specific commit hash we cloned from.

This comment has been minimized.

@zkat

zkat Apr 30, 2017
Contributor

I think we should put the sha512 of the generated tarball here anyway.

It might not be of much use, and the installer can understand enough about this to skip it for git deps if they don't match, but -- this is still useful if folks have already built a tarball locally, cause they'd still be able to do content-addressed lookups this way.

Once we have a tarball packer that can do sequential packs on the same data that have the same checksum, we'll be able to use this more often. We can just avoid verifying the generated tarballs with this, due to those differences.

Furthermore, the hash we cloned from is already in resolved. There's no need for this duplication.


If true, this is the bundled dependency and will be installed by the parent
module. When installing, this module will be extracted from the parent
module during the extract phase, not installed as a separate dependency.

This comment has been minimized.

@zkat

zkat Apr 30, 2017
Contributor

I'd be a little more specific here and say it will be extracted from the root bundler (the first package in the parent chain that doesn't have bundled on it). Bundles can be nested.

## Additional fields / Adding new fields

Installers should ignore any field they aren't aware of. It's not an error
to have additional properities in the shrinkwrap or lock file.

This comment has been minimized.

@Daniel15

Daniel15 May 2, 2017

Small nit: Typo "properities" → "properties"


Installers that want to add new fields should either have one added via RFC
in the npm issue tracker and an accompanying documentation PR, or should prefix
it with the name of their project.

This comment has been minimized.

@Daniel15

Daniel15 May 2, 2017

This should specify the format of the prefix - For example, it could be underscore separated ({project}_{fieldName}) or colon separated ({project}:{fieldName}) or something else altogether.

This comment has been minimized.

@zkat

zkat May 2, 2017
Contributor

What would you suggest?

CSS folks have a pretty "standard" familiar syntax: -brwsr-my-property:

I do like : or :: as separators though. :)

This comment has been minimized.

@Daniel15

Daniel15 May 2, 2017

Honestly, I don't think it matters exactly what the format is, as long as it's documented. Just go with whatever you think looks best 😄 I do think : looks like a nice separator though.

This comment has been minimized.

@zkat

zkat May 2, 2017
Contributor

awesome! 🎉

zkat added a commit that referenced this pull request May 2, 2017
This implements #16441 as well as the auto-save behavior it needs in order to be kept up to date.
@alloy
Copy link

@alloy alloy commented May 2, 2017

Has there been any thought given into making the lockfile shareable with yarn?

@zkochan
Copy link

@zkochan zkochan commented May 30, 2017

I think it would be nice to be able to detect which dependencies cannot be installed on the current system using just the lockfile. Currently, the package.json has to be downloaded in order to check the engines field. Having the engines field in the lockfiles would save us a few redundant HTTP requests.

What do you think?

It may be valuable info for code reviews as well

Mithgol added a commit to Mithgol/node-gamayun that referenced this pull request May 30, 2017
Mithgol added a commit to Mithgol/node-large-split that referenced this pull request May 30, 2017
Mithgol added a commit to Mithgol/node-truncate-escaped-html that referenced this pull request May 30, 2017
Mithgol added a commit to Mithgol/node-twi2fido that referenced this pull request May 30, 2017
@zkat
Copy link
Contributor

@zkat zkat commented May 30, 2017

@zkochan part of the issue is that the definition of "failed optional dep" is literally "tried to install it and it failed". The engines field has always been an FYI and we can't really treat it as normative, lest a lot of other stuff break, I think. /cc @iarna cause I'm pretty sure that's right.

Mithgol added a commit to Mithgol/node-unsquish that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/node-uue that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/node-uue2ipfs that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/node-webbbs that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/npmtree that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/phido that referenced this pull request May 31, 2017
Mithgol added a commit to Mithgol/rss2fido that referenced this pull request Jun 1, 2017
Mithgol added a commit to Mithgol/rules-fe2hpt that referenced this pull request Jun 1, 2017
Mithgol added a commit to Mithgol/shee2arr that referenced this pull request Jun 1, 2017
@zkochan
Copy link

@zkochan zkochan commented Jun 2, 2017

@zkat unless engine-strict is set to true

If set to true, then npm will stubbornly refuse to install (or even consider installing) any package that claims to not be compatible with the current Node.js version.

@isaacs
Copy link
Member

@isaacs isaacs commented Jun 2, 2017

It's well after the time for comments on this particular aspect of the bikeshed, and what I wanted already happened, but .lock as a file extension tells me it's a file system mutex lock. I know yarn and gem don't use it that way, but I've interacted with many programs over the years that do. (For example, this is what a .lock file in an npm <4 cache is.)

The .json extension says "this contains meaningful data".

@gauravmuk
Copy link

@gauravmuk gauravmuk commented Jun 15, 2017

Is there any way that I can not generate package-lock.json every time on npm install with npm@5?

EDIT: got the answer, --no-package-lock


This is a
[subresource integrity](https://w3c.github.io/webappsec/specs/subresourceintegrity/)
value created from the `pacakge.json`. No preprocessing of the

This comment has been minimized.

@saoirse-zee

saoirse-zee Jun 27, 2017

Typo: pacakge --> package :)

@zkat
Copy link
Contributor

@zkat zkat commented Jun 29, 2017

This has been merged 👍

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.