[BUG] install fails with --offline and --global

[ezzieyguywuf]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I am trying to write a gentoo ebuild to install npm packages. In order to do so, I need to perform all network operations prior to calling npm install, thus I am using the --offline option. However, if I pair this with --global, I get an error because npm tries to fetch a dependency (even though I have already installed it).

Expected Behavior

I expect to be able to install packages using both --offline and --global

Steps To Reproduce

The following will reproduce the issue in a sub-directory called 'failure', and show how removing the --global flag fixes the issue in a sub-directory called 'success'

#!/bin/bash

NODE_DIR=node_target
CACHE_DIR=cache_dir

# fetch tarballs. ms is a dependency of debug
wget https://registry.npmjs.org/ms/-/ms-2.1.2.tgz
wget https://registry.npmjs.org/debug/-/debug-4.3.2.tgz

# the ms installation completes without an issue
npm --global --prefix ${PWD}/failure/${NODE_DIR} \
    --cache ${PWD}/failure/${CACHE_DIR} \
    --audit=false --offline --ddd \
    install ./ms-2.1.2.tgz

# the error occurs when trying to install debug globally
npm --global --prefix ${PWD}/failure/${NODE_DIR} \
    --cache ${PWD}/failure/${CACHE_DIR} \
    --audit=false --offline --ddd \
    install ./debug-4.3.2.tgz

# when run without --global, everything is fine
npm --prefix ${PWD}/success/${NODE_DIR} \
    --cache ${PWD}/success/${CACHE_DIR} \
    --audit=false --offline --ddd \
    install ./ms-2.1.2.tgz

npm --prefix ${PWD}/success/${NODE_DIR} \
    --cache ${PWD}/success/${CACHE_DIR} \
    --audit=false --offline --ddd \
    install ./debug-4.3.2.tgz

The error that I see is the following (this is a snippet of the full output):

npm timing idealTree:fixDepFlags Completed in 1ms
npm timing idealTree Completed in 79ms
npm timing command:install Completed in 103ms
npm verb stack Error: request to https://registry.npmjs.org/ms failed: cache
mode is 'only-if-cached' but no cached response is available.
npm verb stack     at cacheFetch
(/usr/lib64/node_modules/npm/node_modules/make-fetch-happen/lib/cache/index.js:12:13)
npm verb stack     at async fetch
(/usr/lib64/node_modules/npm/node_modules/make-fetch-happen/lib/fetch.js:84:7)
npm verb stack     at async Promise.all (index 0)
npm verb stack     at async Arborist.[buildDepStep]
(/usr/lib64/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:973:5)
npm verb stack     at async Arborist.buildIdealTree
(/usr/lib64/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:203:7)
npm verb stack     at async Arborist.reify
(/usr/lib64/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:141:5)
npm verb stack     at async Install.install
(/usr/lib64/node_modules/npm/lib/install.js:150:5)
npm verb cwd /home/wolfie/Program/tmp
npm verb Linux 5.10.60
npm verb argv "/usr/bin/node" "/usr/bin/npm" "--global" "--prefix"
"/home/wolfie/Program/tmp/failure/node_target" "--cache"
"/home/wolfie/Program/tmp/failure/cache_dir" "--audit=false" "--offline" "--ddd"
"install" "./debug-4.3.2.tgz"
npm verb node v16.6.2
npm verb npm  v7.20.3
npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/ms failed: cache mode is
'only-if-cached' but no cached response is available.
npm verb exit 1
npm timing npm Completed in 299ms
npm verb unfinished npm timer reify 1631737242592
npm verb unfinished npm timer reify:loadTrees 1631737242613
npm verb code 1

Environment

• OS: gentoo
• Node: v16.6.2
• npm: 7.20.3

As I've been reviewing the npm code base, I think the following from the arborist dependency provides a clue:

 // when doing a local install, we load everything and figure it all out.
 // when doing a global install, we *only* care about the explicit requests.
  [_loadTrees] (options) {
    process.emit('time', 'reify:loadTrees')
    const bitOpt = {
      ...options,
      complete: this[_packageLockOnly] || this[_dryRun],
    }

I added a bit of debugging to the make-fetch-happen dependency, which is further up the stack chain. Specifically, I was curious about the options parameter passed to the fetch function - I wanted to see if it differed at all with/without the --global flag.

What I found is the following:

const withGlobal = JSON.stringify(options);
const withoutGlobal = JSON.stringify(options);

// inspecting the two, the only difference is that:
withGlobal["pacote-pkg-id"] === "registry:ms";
withoutGlobal["pacote-pkg-id"] === "registry:npm";

I'm not quite sure what to make of this though. The final clue that I found (but again, not quite sure what to make of it) is this comment back in the arborist code:

 // pre-fetch any problem edges, since we'll need these soon
 // if it fails at this point, though, dont' worry because it
 // may well be an optional dep that has gone missing.  it'll
 // fail later anyway. 

I don't understand why this "pre-fetch" is occuring for the --global case only.

[t1m0thyj]

My team is experiencing the same issue, where a TGZ fails to install with the --global and --offline flags, even though it has all dependencies bundled. In our case, the issue only happens on Windows.

This is an important use case for us, to be able to globally install a CLI package without an Internet connection, since some customers are behind a restrictive firewall or proxy.

I've created a minimal sample repo that uses GitHub Actions to demonstrate the issue. The same package is being installed on Windows/Linux/macOS, and you can see in the output of the "Download and Verify" step that all dependencies (including encoding) are bundled, but the install fails on Windows.

[ezzieyguywuf]

I've created a minimal sample repo that uses GitHub Actions

Nice, maybe this will help to gain more visibility.

I've been digging through the npm source code to try to figure out what's going on: as best as I can tell, this has something to do with non---global installs not using "secret" package-lock.json, though I'm not really sure why this makes a difference.

I did find that adding --global-style instead of --global does seem to resolve the issue, but I'm not really sure if this is what I want.

[wraithgar]

The reason for this error is because the tarball is not the only thing that is fetched by npm during install. When you install debug it looks at the package.json and sees that it has a dependency on { ms: 2.1.2 }. npm will then try to fetch the ms packument from the registry to see what that package has for dependencies. It needs the second packument to know where the tarball is. From there it will try to fetch that packument, and it would have a cache hit if possible.

$ npm info ms dist.tarball
https://registry.npmjs.org/ms/-/ms-2.1.3.tgz

If you need ms to be fully cached you have to install it by name, not by tarball. Even if you were to do this, you'd run into your next problem, which is the debug tarball is cached as a file: url. The npm cache works like a web cache, the source url matters. The tgz you have in there already wouldn't be a HIT for the dist.tarball that ms lists.

$ npm info ms dist.tarball
https://registry.npmjs.org/ms/-/ms-2.1.3.tgz

~/D/n/scratch $ npm cache ls --cache ./failure/cache/
pacote:tarball:file:../../../debug-4.3.2.tgz
pacote:tarball:file:../../../ms-2.1.2.tgz
pacote:tarball:file:debug-4.3.2.tgz
pacote:tarball:file:ms-2.1.2.tgz

npm itself has to be used to warm the cache, even if it's by other commands like npm cache add. I don't know where in your chain the network restriction happens, but if you can run other npm commands first you can run npm cache add ms@2.1.2 with the appropriate --prefix and --cache to get the ms packument and tarball in your cache. You could even do the same with debug and skip the manual tarball fetches also.

[wraithgar]

~/D/n/scratch $ rm -rf ./failure/
~/D/n/scratch $ npm cache add debug --prefix ./failure/prefix/ --cache ./failure/cache
~/D/n/scratch $ npm cache add ms@2.1.2 --prefix ./failure/prefix/ --cache ./failure/cache
~/D/n/scratch $ npm --global --prefix ./failure/prefix --cache ./failure/cache --no-audit --offline --ddd install debug
npm verb cli [
npm verb cli   '/Users/wraithgar/.nvm/versions/node/v16.13.2/bin/node',
npm verb cli   '/Users/wraithgar/.nvm/versions/node/v16.13.2/bin/npm',
npm verb cli   '--global',
npm verb cli   '--prefix',
npm verb cli   './failure/prefix',
npm verb cli   '--cache',
npm verb cli   './failure/cache',
npm verb cli   '--no-audit',
npm verb cli   '--offline',
npm verb cli   '--ddd',
npm verb cli   'install',
npm verb cli   'debug'
npm verb cli ]
npm info using npm@8.1.2
npm info using node@v16.13.2
npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 1ms
npm timing config:load:file:/Users/wraithgar/.nvm/versions/node/v16.13.2/lib/node_modules/npm/npmrc Completed in 1ms
npm timing config:load:builtin Completed in 1ms
npm timing config:load:cli Completed in 2ms
npm timing config:load:env Completed in 0ms
npm timing config:load:file:/Users/wraithgar/Development/npm/scratch/failure/prefix/.npmrc Completed in 0ms
npm timing config:load:project Completed in 0ms
npm timing config:load:file:/Users/wraithgar/.npmrc Completed in 2ms
npm timing config:load:user Completed in 2ms
npm timing config:load:file:/Users/wraithgar/Development/npm/scratch/failure/prefix/etc/npmrc Completed in 0ms
npm timing config:load:global Completed in 0ms
npm timing config:load:validate Completed in 0ms
npm timing config:load:credentials Completed in 1ms
npm timing config:load:setEnvs Completed in 1ms
npm timing config:load Completed in 8ms
npm timing npm:load:configload Completed in 9ms
npm timing npm:load:setTitle Completed in 16ms
npm timing npm:load:setupLog Completed in 1ms
npm timing config:load:flatten Completed in 2ms
npm timing npm:load:cleanupLog Completed in 2ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load:projectScope Completed in 1ms
npm timing npm:load Completed in 35ms
npm timing arborist:ctor Completed in 1ms
npm timing idealTree:init Completed in 3ms
npm timing idealTree:userRequests Completed in 1ms
npm sill idealTree buildDeps
npm sill fetch manifest debug@*
npm http fetch GET 200 https://registry.npmjs.org/debug 11ms (cache hit)
npm sill placeDep ROOT debug@4.3.3 OK for:  want: *
npm sill fetch manifest ms@2.1.2
npm http fetch GET 200 https://registry.npmjs.org/ms 4ms (cache hit)
npm timing idealTree:#root Completed in 23ms
npm sill placeDep node_modules/debug ms@2.1.2 OK for: debug@4.3.3 want: 2.1.2
npm timing idealTree:node_modules/debug Completed in 2ms
npm timing idealTree:node_modules/debug/node_modules/ms Completed in 0ms
npm timing idealTree:buildDeps Completed in 26ms
npm timing idealTree:fixDepFlags Completed in 1ms
npm timing idealTree Completed in 32ms
npm timing reify:loadTrees Completed in 33ms
npm timing reify:diffTrees Completed in 0ms
npm sill reify moves {}
npm timing reify:retireShallow Completed in 0ms
npm timing reify:createSparse Completed in 3ms
npm timing reify:loadBundles Completed in 0ms
npm timing reifyNode:node_modules/debug/node_modules/ms Completed in 16ms
npm timing reifyNode:node_modules/debug Completed in 17ms
npm timing reify:unpack Completed in 17ms
npm timing reify:unretire Completed in 0ms
npm timing build:queue Completed in 0ms
npm timing build:deps Completed in 0ms
npm timing build Completed in 1ms
npm timing reify:build Completed in 1ms
npm timing reify:trash Completed in 0ms
npm timing reify Completed in 73ms
npm sill ADD node_modules/debug
npm sill ADD node_modules/debug/node_modules/ms

added 2 packages in 254ms
npm timing command:install Completed in 76ms
npm verb exit 0
npm timing npm Completed in 255ms
npm info ok 

Note the full urls in the tarballs

npm cache ls --cache ./failure/cache/
make-fetch-happen:request-cache:https://registry.npmjs.org/debug
make-fetch-happen:request-cache:https://registry.npmjs.org/debug/-/debug-4.3.3.tgz
make-fetch-happen:request-cache:https://registry.npmjs.org/ms
make-fetch-happen:request-cache:https://registry.npmjs.org/ms/-/ms-2.1.2.tgz
pacote:tarball:debug@
pacote:tarball:ms@2.1.2