idea: install from cache only #1738

Closed
dominictarr opened this Issue Nov 20, 2011 · 14 comments

Comments

Projects
None yet
@dominictarr

add an option to temporarily disable checking the registry for updated packages, and install straight from the cache.

basically, acting as if the request that checks if there is some thing more recent returned 'no', but without making the request.

only if a needed module was in the cache, would npm make a request to the registry.

the benefits that I imagine are greater speed, since there are potentially 0 network round trips.
this could be for applications that needed to constantly install things, like a testing service,
which could update the cache with a separate process listening on the _changes feed.

also, it you could use npm offline, if you didn't need new modules.

@isaacs isaacs closed this in c7d941d Dec 9, 2011

@Raynos

This comment has been minimized.

Show comment Hide comment
@Raynos

Raynos Sep 16, 2012

Contributor

This means npm install module --force would work offline if i've ever installed that module before?

Contributor

Raynos commented Sep 16, 2012

This means npm install module --force would work offline if i've ever installed that module before?

@opichals

This comment has been minimized.

Show comment Hide comment
@opichals

opichals Nov 15, 2012

The --force doesn't seem to work. Moreover if the npmjs.org hostname is not resolvable the npm install takes forever.

So far the only hack I found that makes npm get the package .tgz from its local cache without actually going out to the internet is to run a simple HTTP 304 returning http server and set the to the npm as the registry URL.

# fetch the package and its dependencies into a local cache folder:
npm_config_cache=./npm_cache npm install express

# run the 304 HTTP server
node -e "var http = require('http'); var fs = require('fs'); http.createServer(function (req, res) { res.writeHead(304, {'Content-Type': 'text/plain'}); res.end(); }).listen(9615);" &

# now this will actually install even when the npmjs.org is not resolvable
npm_config_registry=http://localhost:9615/ npm_config_cache=./npm_cache npm install express

# shut the 304 server down
fg
CTRL+C

The --force doesn't seem to work. Moreover if the npmjs.org hostname is not resolvable the npm install takes forever.

So far the only hack I found that makes npm get the package .tgz from its local cache without actually going out to the internet is to run a simple HTTP 304 returning http server and set the to the npm as the registry URL.

# fetch the package and its dependencies into a local cache folder:
npm_config_cache=./npm_cache npm install express

# run the 304 HTTP server
node -e "var http = require('http'); var fs = require('fs'); http.createServer(function (req, res) { res.writeHead(304, {'Content-Type': 'text/plain'}); res.end(); }).listen(9615);" &

# now this will actually install even when the npmjs.org is not resolvable
npm_config_registry=http://localhost:9615/ npm_config_cache=./npm_cache npm install express

# shut the 304 server down
fg
CTRL+C
@isaacs

This comment has been minimized.

Show comment Hide comment
@isaacs

isaacs Nov 16, 2012

Member

I'd be happy to accept a patch that would use the cache if the registry isn't reachable. Part of the problem is that there's no easy way to know if the registry is reachable except to try to reach it, which is going to be slow if you are disconnected.

You can do --no-registry and it'll only use the cache.

Member

isaacs commented Nov 16, 2012

I'd be happy to accept a patch that would use the cache if the registry isn't reachable. Part of the problem is that there's no easy way to know if the registry is reachable except to try to reach it, which is going to be slow if you are disconnected.

You can do --no-registry and it'll only use the cache.

@opichals

This comment has been minimized.

Show comment Hide comment
@opichals

opichals Dec 3, 2012

Oh, thanks for the --no-registry link. I wasn't able to find this in the docs nor by simple grep of the npm sources unfortunately before. Works great!

There is probably an unrelated issue with npm trying to fetch the original registry URL at any point when a dependency is stated as '== x.y.z' instead of just 'x.y.x'. When the package had the == version specifier it was always trying to reach out to the registry URL no matter whether the --no-registry was present or not.

opichals commented Dec 3, 2012

Oh, thanks for the --no-registry link. I wasn't able to find this in the docs nor by simple grep of the npm sources unfortunately before. Works great!

There is probably an unrelated issue with npm trying to fetch the original registry URL at any point when a dependency is stated as '== x.y.z' instead of just 'x.y.x'. When the package had the == version specifier it was always trying to reach out to the registry URL no matter whether the --no-registry was present or not.

@isaacs

This comment has been minimized.

Show comment Hide comment
@isaacs

isaacs Dec 4, 2012

Member

@opichals Get rid of the ==. That's not a valid semver, I don't know why you'd even have that there.

Member

isaacs commented Dec 4, 2012

@opichals Get rid of the ==. That's not a valid semver, I don't know why you'd even have that there.

@opichals

This comment has been minimized.

Show comment Hide comment
@opichals

opichals Dec 5, 2012

I know. npm didn't complain about that not being correct... it just kept fetching the packages even though they were in the cache (the requested package name there was prepended with a space).

opichals commented Dec 5, 2012

I know. npm didn't complain about that not being correct... it just kept fetching the packages even though they were in the cache (the requested package name there was prepended with a space).

@albertnetymk

This comment has been minimized.

Show comment Hide comment
@albertnetymk

albertnetymk Mar 14, 2013

@isaacs "Part of the problem is that there's no easy way to know if the registry is reachable except to try to reach it, which is going to be slow if you are disconnected."

Probably solving the problem completely is very difficult, but maybe we could make it more intelligent by first inspecting whether network interface has valid IP address, as mentioned in here, then really access the registry if so. This won't hit all the cases, but most scenarios are covered, I supposed.

@isaacs "Part of the problem is that there's no easy way to know if the registry is reachable except to try to reach it, which is going to be slow if you are disconnected."

Probably solving the problem completely is very difficult, but maybe we could make it more intelligent by first inspecting whether network interface has valid IP address, as mentioned in here, then really access the registry if so. This won't hit all the cases, but most scenarios are covered, I supposed.

@DTrejo

This comment has been minimized.

Show comment Hide comment
@DTrejo

DTrejo Sep 6, 2013

Contributor

Didn't know about --no-registry and wrote my own thing to install npm packages when offline. woops!
https://github.com/dtrejo/urn

Contributor

DTrejo commented Sep 6, 2013

Didn't know about --no-registry and wrote my own thing to install npm packages when offline. woops!
https://github.com/dtrejo/urn

@luk-

This comment has been minimized.

Show comment Hide comment
@luk-

luk- Sep 6, 2013

Contributor

👍 🍦

Contributor

luk- commented Sep 6, 2013

👍 🍦

@tomyan

This comment has been minimized.

Show comment Hide comment
@tomyan

tomyan Oct 23, 2013

Just managed to track down why --no-registry didn't seem to work with a cache. Turns out somebody else had fixed it:

49fda3a

Just need to upgrade to npm v1.3.11 (in node v0.10.19) to get the fix. Before that fix it was mixing the versions for different packages and thinking it didn't have a package in the cache.

tomyan commented Oct 23, 2013

Just managed to track down why --no-registry didn't seem to work with a cache. Turns out somebody else had fixed it:

49fda3a

Just need to upgrade to npm v1.3.11 (in node v0.10.19) to get the fix. Before that fix it was mixing the versions for different packages and thinking it didn't have a package in the cache.

@DTrejo

This comment has been minimized.

Show comment Hide comment
@DTrejo

DTrejo Oct 26, 2013

Contributor

thanks for the tip @tomyan, I was wondering what was going on

Contributor

DTrejo commented Oct 26, 2013

thanks for the tip @tomyan, I was wondering what was going on

@renan

This comment has been minimized.

Show comment Hide comment
@renan

renan Nov 15, 2013

--no-registry would solve the problem of the registry being down. It works great for development environments, but I am not going to enable it for production environments and risk a deploy (ie: deploy contains a new package or a different package version, which is not cached).

Whenever a package has a pinned version (also when using shrinkwrap) it should first check if the package+version you are trying to install is cached. Use the cached package+version if it exists, otherwise fetches from registry.

This would considerably reduce the hits to the registry and there would be almost none 304 (Not Modified) responses. IMO.

renan commented Nov 15, 2013

--no-registry would solve the problem of the registry being down. It works great for development environments, but I am not going to enable it for production environments and risk a deploy (ie: deploy contains a new package or a different package version, which is not cached).

Whenever a package has a pinned version (also when using shrinkwrap) it should first check if the package+version you are trying to install is cached. Use the cached package+version if it exists, otherwise fetches from registry.

This would considerably reduce the hits to the registry and there would be almost none 304 (Not Modified) responses. IMO.

@mikermcneil

This comment has been minimized.

Show comment Hide comment
@mikermcneil

mikermcneil Dec 16, 2013

The workaround here works for npm v1.3.17:
isaacs#2568 (comment)

The workaround here works for npm v1.3.17:
isaacs#2568 (comment)

@jalcine

This comment has been minimized.

Show comment Hide comment
@jalcine

jalcine May 8, 2014

Does @renan's idea of checking for cached versions apply for shrinkwrap?

jalcine commented May 8, 2014

Does @renan's idea of checking for cached versions apply for shrinkwrap?

@boennemann boennemann referenced this issue in hoodiehq-archive/hoodie-cli Aug 23, 2014

Merged

Install template from cache only #146

@othiym23 othiym23 added this to the cache rewrite milestone Sep 24, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment