Skip to content
This repository has been archived by the owner. It is now read-only.

release: npm@5.0.0 #16244

wants to merge 222 commits into from

release: npm@5.0.0 #16244

wants to merge 222 commits into from


Copy link

@zkat zkat commented Apr 4, 2017

NOTE: This PR is a WIP. npm@5 beta is currently underway. See #16510 for deets.

Wowowowowow npm@5!

This release marks months of hard work for the young, scrappy, and hungry CLI team, and includes some changes we've been hoping to do for literally years. npm@5 takes npm a pretty big step forward, significantly improving its performance in almost all common situations, fixing a bunch of old errors due to the architecture, and just generally making it more robust and fault-tolerant. It comes with changes to make life easier for people doing monorepos, for users who want consistency/security guarantees, and brings semver support to git dependencies. See below for all the deets!

Breaking Changes

  • One binary to rule them all: ./cli.js has been removed in favor of ./bin/npm-cli.js. In case you were doing something with ./cli.js itself. (#12096)

  • Stub file removed (#16204)

  • Existing npm caches will no longer be used: you will have to redownload any cached packages. There is no tool or intention to reuse old caches. (#15666)

  • npm install ./packages/subdir will now create a symlink instead of a regular installation. file://path/to/tarball.tgz will not change -- only directories are symlinked. (#15900)

  • npm will now scold you if you capitalize its name. seriously it will fight you.

  • The "extremely legacy" _token couchToken has been removed. (#12986)

  • npm will --save by default now. Additionally, package-lock.json will be automatically created unless an npm-shrinkwrap.json exists. (#15666)

  • git dependencies support semver (#15308, #15666)

  • git dependencies with prepare scripts will have their devDependencies installed, and npm install run in their directory before being packed.

  • npm cache commands have been rewritten and don't really work anything like they did before. (#15666)

  • --cache-min and --cache-max have been deprecated. (#15666)

  • running npm while offline will no longer insist on retrying network requests (#15666)

  • The contents of ~/.npm have completely changed.

  • A new loglevel, notice, has been added and set as default.

  • package locks no longer exclude optionalDependencies that failed to build. This means package-lock.json and npm-shrinkwrap.json should now be cross-platform. (#15900)

  • shrinkwrap and package-lock no longer exclude devDependencies. You can still prevent their installation with --prod/--only=prod. (#15900)

  • If you generated your package lock against registry A, and you switch to registry B, npm will now try to install the packages from registry B, instead of A. If you want to use different registries for different packages, use scope-specific registries (npm config set @myscope:registry=https://myownregist.ry/packages/). Different registries for different unscoped packages are not supported anymore.

  • shrinkwrap and package-lock no longer warn and exit without saving the lockfile

  • Local tarballs can now only be installed if they have a file extensions .tar, .tar.gz, or .tgz.

Feature Summary

There's a bunch of exciting stuff going in here! You can track all the changes planned for npm@5 on its project tracker.

Installer changes

  • A new, standardised lockfile feature meant for cross-package-manager compatibility (package-lock.json), and a new format and semantics for shrinkwrap. (#16441)

  • --save is no longer necessary. All installs will be saved by default. You can prevent saving with --no-save. Installing optional and dev deps is unchanged: use -D/--save-dev and -O/--save-optional if you want them saved into those fields instead. Note that since npm@3, npm will automatically update npm-shrinkwrap.json when you save: this will also be true for package-lock.json. (#15666)

  • Installing a package directory now ends up creating a symlink and does the Right Thing™ as far as saving to and installing from npm-shrinkwrap.json goes. If you have a monorepo, this should make things much easier to work with, and probably a lot faster too. 😁 (#15900)

  • git dependencies now support semver-based matching: npm install git://^5 (#15308, #15666)

  • git dependencies with prepare scripts will now have their devDependencies installed, and their prepare script executed as if under npm pack.

  • node-gyp now supports node-gyp.cmd on Windows (#14568)

  • npm no longer blasts your screen with the whole installed tree. Instead, you'll see a summary report of the install that is much kinder on your shell real-estate. Specially for large projects. (#15914):

$ npm install
npm added 125, removed 32, updated 148 and moved 5 packages in 5.032s.
  • --parseable and --json now work more consistently across various commands, particularly install and ls.

  • Indentation is now detected and preserved for package.json, package-lock.json, and npm-shrinkwrap.json. If the package lock is missing, it will default to package.json's current indentation.


Cache Rewrite!

We've been talking about rewriting the cache for a loooong time. So here it is. Lots of exciting stuff ahead. The rewrite will also enable some exciting future features, but we'll talk about those when they're actually in the works. #15666 is the main PR for all these changes. Additional PRs/commits are linked inline.

  • package metadata, package download, and caching infrastructure replaced

  • It's a bit faster. Hopefully it will be noticeable. 🤔

  • With the shrinkwrap and package-lock changes, tarballs will be looked up in the cache by content address (and verified with it)

  • Corrupted cache entries will automatically be removed and re-fetched on integrity check failure.

  • npm CLI now supports tarball hashes with any hash function supported by Node.js. That is, it will use sha512 for tarballs from registries that send a sha512 checksum as the tarball hash. Publishing with sha512 is added by npm/npm-registry-client#157 and may be backfilled by the registry for older entries.

  • remote tarball requests are now properly cached. This means that even if you're missing the integrity field in your npm-shrinkwrap.json, npm will be able to install from the cache.

  • Downloads for large packages are streamed in and out of disk. npm is now able to install packages of """any""" size without running out of memory. Support for publishing them is pending (due to registry limitations).

  • Automatic fallback-to-offline mode. npm will seamlessly use your cache if you are offline, or if you lose access to a particular registry (for example, if you can no longer access a private npm repo, or if your git host is unavailable).

  • A new --prefer-offline option will make npm skip any conditional requests (304 checks) for stale cache data, and only hit the network if something is missing from the cache.

  • A new --prefer-online option that will force npm to revalidate cached data (with 304 checks), ignoring any staleness checks, and refreshing the cache with revalidated, fresh data.

  • A new --offline option will force npm to use the cache or exit. It will error with an ENOTCACHED code if anything it tries to install isn't already in the cache.

  • A new npm cache verify command that will garbage collect your cache, reducing disk usage for things you don't need (-handwave-), and will do full integrity verification on both the index and the content. This will also be hooked into npm doctor as part of its larger suite of checking tools.

  • The new cache is very fault tolerant and supports concurrent access.

    • Multiple npm processes will not corrupt a shared cache.
    • Corrupted data will not be installed. Data is checked on both insertion and extraction, and treated as if it were missing if found to be corrupted. I will literally bake you a cookie if you manage to corrupt the cache in such a way that you end up with the wrong data in your installation (installer bugs notwithstanding).
    • npm cache clear is no longer useful for anything except clearing up disk space.
  • Package metadata is cached separately per registry and package type: you can't have package name conflicts between locally-installed packages, private repo packages, and public repo packages. Identical tarball data will still be shared/deduplicated as long as their hashes match.

  • HTTP cache-related headers and features are "fully" (lol) supported for both metadata and tarball requests -- if you have your own registry, you can define your own cache settings the CLI will obey!

  • prepublishOnly now runs before the tarball to publish is created, after prepare has run.

  • ...and more!

For release task list

  • First downstreaming to Node.js 8 (for 5/9)
  • Update docs for new/changes features
    • prepare on git dependency installation
    • package-lock.json vs shrinkwrap
    • Auto-save
    • sha512 on new publishes
  • Land file specifier branch (including idempotent package-lock and many other fixes)
    • Last work on file specifier branch (complete spec tests)
    • details! links to links,
    • details! removing links
  • Improve behavior for shrinkwrap/package-lock
    • warn if both files are there, preferring shrinkwrap
    • Add new fields from spec: created-with, shrinkwrap-version, package-integrity,
    • full new install semantics around dependency fields, based on shrinkwrap version
    • warn if the shrinkwrap is an incompatible version
      • for npm@5
    • update link spec doc to match shrinkwrap
    • Stop generating shrinkwrap/lockfiles off the filesystem -- calculate the tree w/ the installer
  • Make npm update save files as right type: devDeps end up getting saved as deps
  • Leave uninstallable optionals in the tree in such a way that they don't get installed by diff-trees but they do end up in the lock.
  • Ensure npm install --production doesn't remove dev deps from the lock.
  • Add notice and timing log levels directly to npmlog
  • Land patch about running prepare when installing git dependencies.
  • Fix cross-protocol redirects: zkat/make-fetch-happen#26
Copy link

@siddharthkp siddharthkp commented Apr 12, 2017

@zkat Will any of these flags become default?

Some folks bump into addition flags but it doesn't reach everyone (example: --cache-min)
--prefer-offline is something that would make npm installs faster for everyone and probably should be a default.

I understand that it breaks expectations on how people think npm works right now (which is --prefer-offline right now?) But, do you think major version is probably the time to change this expectation?

Copy link
Contributor Author

@zkat zkat commented Apr 12, 2017

@siddharthkp There's no significant speed difference between --prefer-offline and --prefer-online when I actually test it (I'm surprised too). And I mean that -- the two installs on the repo I do my usual benchmarks are within 1 second of each other, for a minute-long install. Those 304 check are very fast, and pacote/make-fetch-happen are both very very fast at what they do when it comes to networking.

I see you mentioned that you think users would expect --prefer-offline to be the default, but I don't think that's true in practice: Many/most users install newer versions of packages, for example, by doing npm i -S pkg. In this case, using --prefer-offline will not upgrade anything: latest is already locally cached. pacote.manifest() will be satisfied with that, and the user will receive a very stale version of the package manifest. The intention behind --prefer-offline is to give users an option to reduce network usage in-general, if they're aware of the consequences of doing do.

Now, if you're looking for a speedup, the biggest one you can get with npm@5 is to use npm-shrinkwrap.json -- as of this patch, shrinkwrap will include content addresses for all package data, which means that as long as you have a warm cache, the network will not be used at all, and the entire package manifest phase will be completely skipped (because we don't need manifests if we have a shrinkwrap). The speedup from that is often on the order of 2x a non-shrinkwrap cached --prefer-offline install.

So, no, I probably won't use npm@5 to make --prefer-offline the default, since I'm not sure there's a reason to, in practice. Making npm-shrinkwrap be a default, though... that's in the works. Maybe not for npm@5 but I'm pushing for it for sure.

Copy link

@siddharthkp siddharthkp commented Apr 12, 2017

There's no significant speed difference between --prefer-offline and --prefer-online when I actually test it (I'm surprised too).

What were the network conditions like in your benchmarks?

I had a conversation with @iarna a while back (npm 3) where she educated me about the network requests to the registry even when using cache + the effect of latency on it. (Significant for emerging nations - prominently 2G and latency heavy nations - Australia and such)

Are you saying pacote nullifies that effect? (or at least makes it not noticeable) because that would be 💯

I see you mentioned that you think users would expect --prefer-offline to be the default

Ouch! My bad, I meant --prefer-online. Honest mistake.

The speedup from that is often on the order of 2x a non-shrinkwrap cached --prefer-offline install.

This is another interesting part 🤔
Is this because npm still needs to resolve the dependency tree with --prefer-offline which means additional network requests? Not the same with shrinkwrap.

Making npm-shrinkwrap be a default, though... that's in the works.

💯 Is there a github issue that I can upvote for this?

Copy link
Contributor Author

@zkat zkat commented Apr 12, 2017

What were the network conditions like in your benchmarks?

I'm on a Bay Area network, which is admittedly absurdly good. If the difference is so small now, though, I imagine it wouldn't be so big: even at 1/10th the speed, that would still be barely 10 seconds added (to a 60-second installation).

Are you saying pacote nullifies that effect? (or at least makes it not noticeable) because that would be 💯

I really can't say that for sure right now. tbh, the difference is so surprisingly small that I think I'm gonna have to spend some time crawling through tests and making sure I don't have some bug. That code definitely has tests attached, though...

Ouch! My bad, I meant --prefer-online. Honest mistake.

Ah. As far as this one goes, pacote obeys cache age settings from a given registry. This is generally happy news for all involved (if we hit the registry less without degrading user experience, that's a win!), and those settings tend to be fairly low: the main npm registry sets Cache-Control: max-age=300, which is in seconds (5 minutes). After those 5 minutes, we'll do the 304 checks again. This seems reasonable to me, specially with the --prefer-offline escape hatch. Anyway, no worries. They're kinda easy to get confused 😁

I would care more if it weren't for shrinkwrap being the main target of optimization, which brings me to...

Is this because npm still needs to resolve the dependency tree

Pretty much this. If we don't have the exact hashes for the tarballs we want to end up installing, npm needs to repeat the process of lookup by name --> resolve version --> get shasum + dependency names from version --> repeat for all recursive dependencies. That's a pretty heavy process.

💯 Is there a github issue that I can upvote for this?

Not quite yet. There's a github project for npm@5 and one of my TODOs is to write a proposal for how shrinkwrap-by-default would work. From what I've talked with @iarna, though, we need to fix other stuff (like npm update) before we even consider this. So, it'll probably be specced out after npm@5 is out and we're ready to start thinking about npm@6.

Copy link

@siddharthkp siddharthkp commented Apr 12, 2017

If shrinkwrap by default is likely to come in npm6, does it make sense to make --prefer-offline a default in npm5?
Or is that going in the wrong direction?

Copy link

@viktor-ku viktor-ku commented Apr 12, 2017

Any reasons why yarn is so fast? I really expect npm@5 to be as fast as yarn. How can I help?

Copy link

@siddharthkp siddharthkp commented Apr 12, 2017

@Kuroljov With npm-shrinkwrap, npm will be in serious competition with yarn.

@zkat Can we start playing around with that somewhere? (Can contribute to your benchmarks?)

Copy link
Contributor Author

@zkat zkat commented Apr 12, 2017

@Kuroljov This release addresses a lot of performance issues. Right now, npm@5 installs are running anywhere between 2x-5x faster than npm@4. There's a few optimizations left that I don't know if we'll have ready by npm@5, but that number is bound to get significantly bigger (as in, 30%+ from one of those changes)

Some sample benchmarks:

react boilerplate install (including lifecycle scripts):
npm@4 i (no cache): 147.23s
npm@4 i (warm cache): 145.70s
npm@5 i (no cache): 74.30s
npm@5 i (warm cache): 70.85s
npm@5 i (no cache, no shrinkwrap): 99.83s
~/code/npm-website $ npm i --ignore-scripts
warm cache:
npm@4.4.0  - 150s
npm@5.0.0 - 52s
npm@5.0.0 w/ shrinkwrap   - 38s
Copy link
Contributor Author

@zkat zkat commented Apr 12, 2017

@siddharthkp expect a beta release of npm@5 in the coming weeks. I can't give you the exact date, but I'm currently in the process of reviving npm's massive test suite after such a big change and fixing all my little messups along the way.

What I can tell you is that if you make and faster and better, it'll be directly reflected on the performance of npm@5. Once gets its @2 release (probably shortly before npm@beta), the same will apply to that. Those first two projects are relatively stable and usable, though, and I think people can definitely already start contributing significantly to them. There's also but I don't think that's much of a bottleneck at all rn.

I'll make an announcement somewhere dev-oriented once npm@beta is out, cause I definitely want early adopters to help me find all the things that broke during the rewrite -- because it's a big one.

@iarna iarna force-pushed the release-next-5 branch from b1f6b87 to 8bcef6a Apr 14, 2017
Copy link
Contributor Author

@zkat zkat commented Apr 20, 2017

@siddharthkp coming back to this: what are the network conditions where you're at? (since you asked). I'd like to get a better, more realistic perspective on how npm5 handles slow and/or unstable networks and special configurations like proxies, which is all fairly hard to simulate unless you've been deep in that situation, imo. There's now a (semi-closed) beta of npm5 that you can install with npm i -g 'npm/npm#release-beta-5 that will give you an npm5 binary that lives alongside your standard installation. It would be super awesome if folks who are willing to look into weirdness tried it out and gave feedback. I very much want it to be a smooth, seamless experience even in bad network conditions: the worst thing that should happen on unreliable networks is that your download will be slightly slower. Everything else should Just Work™.

iarna and others added 19 commits May 25, 2017
Previously we were doing this so we could load relationships, but now we can
simply request no fake children.  This forces a fetch-metadata phase for the
lockfile, but does not actually install all the modules.
This can happen when, for example, removing a module from your project that
isn't actually in your node_modules at the moment.
It used to be that we did this check in `is-extraneous` but it turns out to
be much cleaner to keep that pure.  It let's us use `is-extraneous` for
things like pruning.
@zkat zkat added the npm5 label May 26, 2017
@zkat zkat force-pushed the release-next-5 branch from 8688036 to 7903602 May 26, 2017
Copy link
Contributor Author

@zkat zkat commented May 27, 2017

This was tagged and released yesterday as npm@5 🎉 thanks everyone!

@zkat zkat closed this May 27, 2017
@zkat zkat deleted the release-next-5 branch May 27, 2017
@zkat zkat removed the in-progress label May 27, 2017

This comment has been minimized.

Copy link

@palmerj3 palmerj3 commented on 0cc5517 May 27, 2017

@iarna thank you for taking care of this!! I was struggling for a bit to get my PR working properly... one question though - it appears that this would never calculate the tree if you run npm with "--ignore-scripts". Is that the case?

Anyway - it's great to see how you approached this and I'll definitely learn from it. Cheers.

Copy link

@olegkorol olegkorol commented Jun 12, 2017


what is the new format of npm-shrinkwrap.json?
After updating npm to v. 5 it won't npm install anymore...

Thanks in advance!


This comment has been minimized.

Copy link

@dotchev dotchev commented on lib/install.js in b97ef35 Aug 25, 2017

What is the reason for this? Who else could install those packages and why not reuse them?
For example we use a private npm registry and don't want to reveal its location in _resolved property in package.json. See #17934.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.