feat(cache): rewrite package fetching and caching on top of pacote

Fixes: #2568
Fixes: #2649
Fixes: #3141
Fixes: #4042
Fixes: #4652
Fixes: #5357
Fixes: #5509
Fixes: #5622
Fixes: #5941

All fetching-related networking is now done through pacote, and
the old cache has been entirely replaced by a cacache-based one.

Features:

* npm now supports a variety of hash algorithms for tarball storage. On registries that support it, npm is able to use sha512sum for verification.

* An `integrity` field has been added to `npm-shrinkwrap.json`.

* Package integrity will be fully verified on both cache insert and extraction -- if npm installs something, it's going to be exactly what you downloaded, byte-for-byte, or it will fail.

* If `npm-shrinkwrap.json` is used, npm will bypass checking package manifests and go straight to the tarball, fetching it by content address if locally cached.

* Checksum integrity failures will now retry downloading on error, instead of failing on a single check.

* A new npm command, `npm cache verify`, can now be used to verify and garbage collect your local cache.

* npm now supports arbitrarily large tarball downloads: tarballs will no longer be loaded entirely into memory before extraction.

* packages whose names only differ in casing, and packages from different sources/registries/etc will now correctly be cached separately from each other.

* Some performance improvements.

* Improved fetch retry logic will try harder to download your packages.

BREAKING CHANGE: many shrinkwrap and cache-related things have changed.

* Previously-created caches will no longer be used. They will be left in place, but data will need to be re-cached. There is no facility for rebuilding a cache based on an existing one.

* `npm cache ls` has been removed for now

* `npm cache rm` now always removes the entire cache. There is no granular removal available for now.

* git dependencies can now use semver resolution using `#semver:^1.2.3`

* `--cache-min` and `--cache-max` have been deprecated. Use `--offline`, `--prefer-offline`, and `--prefer-online instead. `--cache-min=9999+` and `--cache-max=0` have been aliased to `--prefer-offline` and `--prefer-online`, respectively.

* npm will now obey HTTP caching headers sent from registries and other remote HTTP hosts, and will use standard HTTP caching rules for its local cache.

* `prepublishOnly` now runs *before* packing the tarball.

* npm no longer supports node@<4.

doc/misc/npm-config.md

@@ -234,22 +234,18 @@ Number of ms to wait for cache lock files to expire.
 * Default: Infinity
 * Type: Number
 
-The maximum time (in seconds) to keep items in the registry cache before
-re-checking against the registry.
+**DEPRECATED**: This option has been deprecated in favor of `--prefer-online`.
 
-Note that no purging is done unless the `npm cache clean` command is
-explicitly used, and that only GET requests use the cache.
+`--cache-max=0` is an alias for `--prefer-online`.
 
 ### cache-min
 
 * Default: 10
 * Type: Number
 
-The minimum time (in seconds) to keep items in the registry cache before
-re-checking against the registry.
+**DEPRECATED**: This option has been deprecated in favor of `--prefer-offline`.
 
-Note that no purging is done unless the `npm cache clean` command is
-explicitly used, and that only GET requests use the cache.
+`--cache-min=9999 (or bigger)` is an alias for `--prefer-offline`.
 
 ### cert
 
@@ -636,6 +632,14 @@ The registry you want to send cli metrics to if `send-metrics` is true.
 
 The node version to use when checking a package's `engines` map.
 
+### offline
+
+* Default: false
+* Type: Boolean
+
+Force offline mode: no network requests will be done during install. To allow
+the CLI to fill in missing cache data, see `--prefer-offline`.
+
 ### onload-script
 
 * Default: false
@@ -679,6 +683,24 @@ process is not aborted.
 Output parseable results from commands that write to
 standard output. For `npm search`, this will be tab-separated table format.
 
+### prefer-offline
+
+* Default: false
+* Type: Boolean
+
+If true, staleness checks for cached data will be bypassed, but missing data
+will be requested from the server. To force full offline mode, use `--offline`.
+
+This option is effectively equivalent to `--cache-min=9999999`.
+
+### prefer-online
+
+* Default: false
+* Type: Boolean
+
+If true, staleness checks for cached data will be forced, making the CLI look
+for updates immediately even for fresh package data.
+
 ### prefix
 
 * Default: see npm-folders(5)