New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yarn fails to install Electron from the "Offline mirror" folder #3590

Open
manuq opened this Issue Jun 6, 2017 · 13 comments

Comments

Projects
None yet
@manuq

manuq commented Jun 6, 2017

Do you want to request a feature or report a bug?
Bug.

What is the current behavior?
Yarn fails to install Electron from the "Offline mirror" folder. Fails with "Can't make a request in offline mode".

If the current behavior is a bug, please provide the steps to reproduce.

$ node yarn-0.24.6.js config set yarn-offline-mirror ./npm-packages-offline-cache
$ node yarn-0.24.6.js config set yarn-offline-mirror-pruning true
$ node yarn-0.24.6.js init
$ node yarn-0.24.6.js add electron@1.6.10 --dev
$ node yarn-0.24.6.js install

# WORKS if I just remove node_modules/
$ rm -rf node_modules/
$ node yarn-0.24.6.js install --offline

# FAILS if I also clean the cache
$ rm -rf node_modules/
$ node yarn-0.24.6.js cache clean
$ node yarn-0.24.6.js install --offline

The exception is:

$ node yarn-0.24.6.js install --offline
yarn install v0.24.6
[1/4] Resolving packages...
[2/4] Fetching packages...
error Can't make a request in offline mode
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

This is my .yarnrc:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


lastUpdateCheck 1496769371806
yarn-offline-mirror "./npm-offline-cache"
yarn-offline-mirror-pruning true

What is the expected behavior?
According to this blog post, yarn install --offline should work after cleaning the yarn cache. But with Electron, it tries to fetch some package. Adding --verbose doen't help, it does not show what package is trying to fetch.

Please mention your node.js, yarn and operating system version.

  • node v6.10.3
  • yarn 0.24.6
  • GNU/Linux x86_64 Fedora 25
@kavi87

This comment has been minimized.

Show comment
Hide comment
@kavi87

kavi87 Jun 7, 2017

I have the same error with typescript @types definitions that are not added to the offline mirror. If I remove the node_modules, turn off wifi and install --offline everything work but after clearing the cache I get the same error "Can't make a request in offline mode". I'm using Yarn 0.26.

@manuq Yarn 0.26 will log the name of package it failed to fetch now.

kavi87 commented Jun 7, 2017

I have the same error with typescript @types definitions that are not added to the offline mirror. If I remove the node_modules, turn off wifi and install --offline everything work but after clearing the cache I get the same error "Can't make a request in offline mode". I'm using Yarn 0.26.

@manuq Yarn 0.26 will log the name of package it failed to fetch now.

@brianhorst

This comment has been minimized.

Show comment
Hide comment
@brianhorst

brianhorst Jun 7, 2017

I have run into this too. There is an issue with pruning true and scoped dependencies where they are not saved in the offline cache. Take a look at #3509.

I have run into this too. There is an issue with pruning true and scoped dependencies where they are not saved in the offline cache. Take a look at #3509.

@kavi87

This comment has been minimized.

Show comment
Hide comment
@kavi87

kavi87 Jun 7, 2017

@brianhorst Thanks, removing yarn-offline-mirror-pruning true did the trick.

kavi87 commented Jun 7, 2017

@brianhorst Thanks, removing yarn-offline-mirror-pruning true did the trick.

@gunta987

This comment has been minimized.

Show comment
Hide comment
@gunta987

gunta987 Jun 8, 2017

I think the issue may be with packages beginning with '@' not matching the lock file. The same thing happens with the packages installed by react create app: the package called "@timer/detect-port@1.1.3" in the lock file corresponds to the offline mirror file @timer-detect-port-1.1.3.tgz (note the hyphen in place of the slash). At the very end of the online install process, the tgz file is removed from the offline mirror folder.

gunta987 commented Jun 8, 2017

I think the issue may be with packages beginning with '@' not matching the lock file. The same thing happens with the packages installed by react create app: the package called "@timer/detect-port@1.1.3" in the lock file corresponds to the offline mirror file @timer-detect-port-1.1.3.tgz (note the hyphen in place of the slash). At the very end of the online install process, the tgz file is removed from the offline mirror folder.

@manuq

This comment has been minimized.

Show comment
Hide comment
@manuq

manuq Jun 8, 2017

  • Running Yarn install --verbose and without the --offline, I can see the fetch is for Node, the only package in my yarn.lock that has a @types. So @gunta987, worth checking if the issue is general to all packages starting with @.
[2/4] Fetching packages...
verbose 0.8 Performing "GET" request to "https://registry.yarnpkg.com/@types/node/-/node-7.0.29.tgz".
  • @brianhorst, @kavi87 I can confirm that removing yarn-offline-mirror-pruning true from .yarnc does the trick, and makes @types-node-7.0.29.tgz file appear in my "Offline mirror" folder. This is for Yarn v0.24.6

  • Yarn v0.26.1 doesn't have the issue described. So I'm moving my workflow to this version.

manuq commented Jun 8, 2017

  • Running Yarn install --verbose and without the --offline, I can see the fetch is for Node, the only package in my yarn.lock that has a @types. So @gunta987, worth checking if the issue is general to all packages starting with @.
[2/4] Fetching packages...
verbose 0.8 Performing "GET" request to "https://registry.yarnpkg.com/@types/node/-/node-7.0.29.tgz".
  • @brianhorst, @kavi87 I can confirm that removing yarn-offline-mirror-pruning true from .yarnc does the trick, and makes @types-node-7.0.29.tgz file appear in my "Offline mirror" folder. This is for Yarn v0.24.6

  • Yarn v0.26.1 doesn't have the issue described. So I'm moving my workflow to this version.

@rally25rs

This comment has been minimized.

Show comment
Hide comment
@rally25rs

rally25rs Jun 9, 2017

Contributor

Just to confirm, are we in agreement that this is fixed in 0.26.1? If so, can we close this issue since it's already fixed?

Contributor

rally25rs commented Jun 9, 2017

Just to confirm, are we in agreement that this is fixed in 0.26.1? If so, can we close this issue since it's already fixed?

@rally25rs rally25rs added the triaged label Jun 9, 2017

manuq added a commit to manuq/org.sugarizer.Sugarizer that referenced this issue Jun 9, 2017

Finally flatpack-builder build step works offline
Only for Linux64 for now, but more coming.

- Electron version fixed in package.json
- Electron binary added to default 'electron cache' dir (only x64 for now)
- Yarn 'offline mirror' issue is addressed
  yarnpkg/yarn#3590
@subyraman

This comment has been minimized.

Show comment
Hide comment
@subyraman

subyraman Aug 25, 2017

This is still happening for me.

sraman@sraman: bar$ yarn add jsdom@11.0.0 --offline
yarn add v0.27.5
info No lockfile found.
[1/4] Resolving packages...
error Couldn't find any versions for "@types/node" that matches "^6.0.46" in our cache. Possible versions: ""
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

It is definitely available in my cache.

sraman@sraman: bar$ yarn cache ls | grep "@types"
@types/node                                          6.0.78                      npm      https://registry.yarnpkg.com/@types/node/-/node-6.0.78.tgz#5d4a3f579c1524e01ee21bf474e6fba09198f470
@types/node                                          6.0.87                      npm      https://registry.yarnpkg.com/@types/node/-/node-6.0.87.tgz#5ab5774f8351a33a935099fa6be850aa0b0ad564
@types/node                                          6.0.88                      npm      https://registry.yarnpkg.com/@types/node/-/node-6.0.88.tgz#f618f11a944f6a18d92b5c472028728a3e3d4b66

This is still happening for me.

sraman@sraman: bar$ yarn add jsdom@11.0.0 --offline
yarn add v0.27.5
info No lockfile found.
[1/4] Resolving packages...
error Couldn't find any versions for "@types/node" that matches "^6.0.46" in our cache. Possible versions: ""
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

It is definitely available in my cache.

sraman@sraman: bar$ yarn cache ls | grep "@types"
@types/node                                          6.0.78                      npm      https://registry.yarnpkg.com/@types/node/-/node-6.0.78.tgz#5d4a3f579c1524e01ee21bf474e6fba09198f470
@types/node                                          6.0.87                      npm      https://registry.yarnpkg.com/@types/node/-/node-6.0.87.tgz#5ab5774f8351a33a935099fa6be850aa0b0ad564
@types/node                                          6.0.88                      npm      https://registry.yarnpkg.com/@types/node/-/node-6.0.88.tgz#f618f11a944f6a18d92b5c472028728a3e3d4b66
@3DEsprit

This comment has been minimized.

Show comment
Hide comment
@3DEsprit

3DEsprit Dec 17, 2017

So great to see this resolved!

I remember trying to find an answer to this earlier this year with not resolve, and ended up switching back to npm. Since there was a constant back and forth from all three projects of who's module was causing the issue, there wasn't an answer (especially not this one). I no longer feel dirty being able to use Yarn again.

I did find that I continued to have this issue with 1.3.2, even with the node_modules removed. Once I cleared the cache, the install worked without a hitch and both the yarn-offline-mirror and yarn-offline-mirror-pruning true setting set.

So great to see this resolved!

I remember trying to find an answer to this earlier this year with not resolve, and ended up switching back to npm. Since there was a constant back and forth from all three projects of who's module was causing the issue, there wasn't an answer (especially not this one). I no longer feel dirty being able to use Yarn again.

I did find that I continued to have this issue with 1.3.2, even with the node_modules removed. Once I cleared the cache, the install worked without a hitch and both the yarn-offline-mirror and yarn-offline-mirror-pruning true setting set.

@kappa-gooner

This comment has been minimized.

Show comment
Hide comment
@kappa-gooner

kappa-gooner Jan 8, 2018

I still have this issue with

yarn --version
1.3.2
npm --version
5.5.1

The problem is caused by the cypress package. I've tried clearing cache, modifying my yarn-offline-mirror-pruning setting; to no effect. I've a custom registry set though; not sure if it affects anything here.

[2/4] Fetching packages...
verbose 2.45 Error: Can't make a request in offline mode ("https://art-bobcat.autodesk.com:443/artifactory/api/npm/autodesk-npm-virtual/@cypress/listr-verbose-renderer/-/listr-verbose-renderer-0.4.1.tgz")
    at RequestManager.request (/usr/local/lib/node_modules/yarn/lib/cli.js:61929:59)
    at NpmRegistry.request (/usr/local/lib/node_modules/yarn/lib/cli.js:30362:32)
    at /usr/local/lib/node_modules/yarn/lib/cli.js:59335:33
    at Generator.next (<anonymous>)
    at step (/usr/local/lib/node_modules/yarn/lib/cli.js:92:30)
    at /usr/local/lib/node_modules/yarn/lib/cli.js:110:14
    at new Promise (<anonymous>)
    at new F (/usr/local/lib/node_modules/yarn/lib/cli.js:29389:28)
    at /usr/local/lib/node_modules/yarn/lib/cli.js:89:12
    at TarballFetcher.fetchFromExternal (/usr/local/lib/node_modules/yarn/lib/cli.js:59395:7)
error Can't make a request in offline mode ("https://art-bobcat.autodesk.com:443/artifactory/api/npm/autodesk-npm-virtual/@cypress/listr-verbose-renderer/-/listr-verbose-renderer-0.4.1.tgz")
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

kappa-gooner commented Jan 8, 2018

I still have this issue with

yarn --version
1.3.2
npm --version
5.5.1

The problem is caused by the cypress package. I've tried clearing cache, modifying my yarn-offline-mirror-pruning setting; to no effect. I've a custom registry set though; not sure if it affects anything here.

[2/4] Fetching packages...
verbose 2.45 Error: Can't make a request in offline mode ("https://art-bobcat.autodesk.com:443/artifactory/api/npm/autodesk-npm-virtual/@cypress/listr-verbose-renderer/-/listr-verbose-renderer-0.4.1.tgz")
    at RequestManager.request (/usr/local/lib/node_modules/yarn/lib/cli.js:61929:59)
    at NpmRegistry.request (/usr/local/lib/node_modules/yarn/lib/cli.js:30362:32)
    at /usr/local/lib/node_modules/yarn/lib/cli.js:59335:33
    at Generator.next (<anonymous>)
    at step (/usr/local/lib/node_modules/yarn/lib/cli.js:92:30)
    at /usr/local/lib/node_modules/yarn/lib/cli.js:110:14
    at new Promise (<anonymous>)
    at new F (/usr/local/lib/node_modules/yarn/lib/cli.js:29389:28)
    at /usr/local/lib/node_modules/yarn/lib/cli.js:89:12
    at TarballFetcher.fetchFromExternal (/usr/local/lib/node_modules/yarn/lib/cli.js:59395:7)
error Can't make a request in offline mode ("https://art-bobcat.autodesk.com:443/artifactory/api/npm/autodesk-npm-virtual/@cypress/listr-verbose-renderer/-/listr-verbose-renderer-0.4.1.tgz")
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
@lxe

This comment has been minimized.

Show comment
Hide comment
@lxe

lxe Feb 27, 2018

I think this seems to be happening for all (some?) scoped package tarballs.

lxe commented Feb 27, 2018

I think this seems to be happening for all (some?) scoped package tarballs.

@erickj

This comment has been minimized.

Show comment
Hide comment
@erickj

erickj Mar 19, 2018

yarn offline mirrors appear to be completely broken, at least according to the instructions here https://yarnpkg.com/blog/2016/11/24/offline-mirror/ which appear to be the official documentation

there are 2 different bugs AFAICT:

bug 1. resolving scoped types when the yarn cache is populated. to repro, follow the steps in the offline-mirror blog post, DO NOT clean the cache, add (e.g.) rollup as a dependency, which depends on @types/acorn, run yarn install --offline and see this error:

$ yarn install --offline
yarn install v1.5.1
info No lockfile found.
[1/4] Resolving packages...
error An unexpected error occurred: "Couldn't find package \"@types/acorn@^4.0.3\" required by \"rollup@^0.57.1\" on the \"npm\" registry.".
info If you think this is a bug, please open a bug report with the information provided in "/tmp/npm-test/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Error: Couldn't find package "@types/estree@0.0.38" required by "is-reference@^1.1.0" on the "npm" registry.
    at new MessageError (/home/erick/.npm_packages/lib/node_modules/yarn/lib/cli.js:186:110)
    at PackageRequest.<anonymous> (/home/erick/.npm_packages/lib/node_modules/yarn/lib/cli.js:36274:17)
    at Generator.throw (<anonymous>)
    at step (/home/erick/.npm_packages/lib/node_modules/yarn/lib/cli.js:98:30)
    at /home/erick/.npm_packages/lib/node_modules/yarn/lib/cli.js:111:13
    at <anonymous>

bug 2. locating any offline packages after teh cache has been cleaned: again follow the steps in the offline-mirror blog post, add lodash as a simple dependency with no transitive dependencies, this time do a yarn cache clean before attempting the offline install. result:

$ yarn install --offline
yarn install v1.5.1
info No lockfile found.
[1/4] Resolving packages...
error An unexpected error occurred: "Couldn't find any versions for \"lodash\" that matches \"^4.17.5\" in our cache (possible versions are \"\"). This is usually caused by a missing entry in the lockfile, running Yarn without the --offline flag may help fix this issue.".
info If you think this is a bug, please open a bug report with the information provided in "/tmp/npm-test/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

this is with yarn 1.5.1:

$ yarn config list
yarn config v1.5.1
info yarn config
{ 'version-tag-prefix': 'v',
  'version-git-tag': true,
  'version-git-sign': false,
  'version-git-message': 'v%s',
  'init-version': '1.0.0',
  'init-license': 'MIT',
  'save-prefix': '^',
  'ignore-scripts': false,
  'ignore-optional': false,
  registry: 'https://registry.yarnpkg.com',
  'strict-ssl': true,
  'user-agent': 'yarn/1.5.1 npm/? node/v8.9.1 linux x64',
  lastUpdateCheck: 1521455361620,
  'yarn-offline-mirror': '/home/erick/npm-packages-offline-cache',
  'yarn-offline-mirror-pruning': false }
info npm config
{ prefix: '/home/erick/.npm_packages' }
Done in 0.04s.

erickj commented Mar 19, 2018

yarn offline mirrors appear to be completely broken, at least according to the instructions here https://yarnpkg.com/blog/2016/11/24/offline-mirror/ which appear to be the official documentation

there are 2 different bugs AFAICT:

bug 1. resolving scoped types when the yarn cache is populated. to repro, follow the steps in the offline-mirror blog post, DO NOT clean the cache, add (e.g.) rollup as a dependency, which depends on @types/acorn, run yarn install --offline and see this error:

$ yarn install --offline
yarn install v1.5.1
info No lockfile found.
[1/4] Resolving packages...
error An unexpected error occurred: "Couldn't find package \"@types/acorn@^4.0.3\" required by \"rollup@^0.57.1\" on the \"npm\" registry.".
info If you think this is a bug, please open a bug report with the information provided in "/tmp/npm-test/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Error: Couldn't find package "@types/estree@0.0.38" required by "is-reference@^1.1.0" on the "npm" registry.
    at new MessageError (/home/erick/.npm_packages/lib/node_modules/yarn/lib/cli.js:186:110)
    at PackageRequest.<anonymous> (/home/erick/.npm_packages/lib/node_modules/yarn/lib/cli.js:36274:17)
    at Generator.throw (<anonymous>)
    at step (/home/erick/.npm_packages/lib/node_modules/yarn/lib/cli.js:98:30)
    at /home/erick/.npm_packages/lib/node_modules/yarn/lib/cli.js:111:13
    at <anonymous>

bug 2. locating any offline packages after teh cache has been cleaned: again follow the steps in the offline-mirror blog post, add lodash as a simple dependency with no transitive dependencies, this time do a yarn cache clean before attempting the offline install. result:

$ yarn install --offline
yarn install v1.5.1
info No lockfile found.
[1/4] Resolving packages...
error An unexpected error occurred: "Couldn't find any versions for \"lodash\" that matches \"^4.17.5\" in our cache (possible versions are \"\"). This is usually caused by a missing entry in the lockfile, running Yarn without the --offline flag may help fix this issue.".
info If you think this is a bug, please open a bug report with the information provided in "/tmp/npm-test/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

this is with yarn 1.5.1:

$ yarn config list
yarn config v1.5.1
info yarn config
{ 'version-tag-prefix': 'v',
  'version-git-tag': true,
  'version-git-sign': false,
  'version-git-message': 'v%s',
  'init-version': '1.0.0',
  'init-license': 'MIT',
  'save-prefix': '^',
  'ignore-scripts': false,
  'ignore-optional': false,
  registry: 'https://registry.yarnpkg.com',
  'strict-ssl': true,
  'user-agent': 'yarn/1.5.1 npm/? node/v8.9.1 linux x64',
  lastUpdateCheck: 1521455361620,
  'yarn-offline-mirror': '/home/erick/npm-packages-offline-cache',
  'yarn-offline-mirror-pruning': false }
info npm config
{ prefix: '/home/erick/.npm_packages' }
Done in 0.04s.
@rally25rs

This comment has been minimized.

Show comment
Hide comment
@rally25rs

rally25rs Mar 19, 2018

Contributor

@erickj I can't seem to reproduce what you are experiencing. (see edit below)

~/Projects/yarn-test 🐒   yarn init -y
yarn init v1.5.1
warning The yes flag has been set. This will automatically answer yes to all questions, which may have security implications.
success Saved package.json
✨  Done in 0.04s.

~/Projects/yarn-test 🐒   cat .yarnrc
yarn-offline-mirror "./npm-packages-offline-cache"
yarn-offline-mirror-pruning true

~/Projects/yarn-test 🐒   yarn add rollup
yarn add v1.5.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 48 new dependencies.
info Direct dependencies
└─ rollup@0.57.1
info All dependencies
├─ @types/acorn@4.0.3
├─ @types/estree@0.0.38
├─ acorn-dynamic-import@3.0.0
├─ acorn@5.5.3
├─ arr-diff@2.0.0
├─ arr-flatten@1.1.0
├─ array-unique@0.2.1
├─ braces@1.8.5
├─ date-time@2.1.0
├─ estree-walker@0.3.1
├─ expand-brackets@0.1.5
├─ expand-range@1.8.2
├─ extglob@0.3.2
├─ filename-regex@2.0.1
├─ fill-range@2.2.3
├─ for-in@1.0.2
├─ for-own@0.1.5
├─ glob-base@0.3.0
├─ glob-parent@2.0.0
├─ irregular-plurals@1.4.0
├─ is-dotfile@1.0.3
├─ is-equal-shallow@0.1.3
├─ is-extendable@0.1.1
├─ is-number@2.1.0
├─ is-posix-bracket@0.1.1
├─ is-primitive@2.0.0
├─ is-reference@1.1.0
├─ isarray@1.0.0
├─ isobject@2.1.0
├─ locate-character@2.0.5
├─ micromatch@2.3.11
├─ normalize-path@2.1.1
├─ object.omit@2.0.1
├─ parse-glob@3.0.4
├─ parse-ms@1.0.1
├─ plur@2.1.2
├─ preserve@0.2.0
├─ pretty-ms@3.1.0
├─ randomatic@1.1.7
├─ regex-cache@0.4.4
├─ remove-trailing-separator@1.1.0
├─ repeat-string@1.6.1
├─ require-relative@0.8.7
├─ rollup-pluginutils@2.0.1
├─ rollup@0.57.1
├─ signal-exit@3.0.2
├─ sourcemap-codec@1.4.1
└─ time-zone@1.0.0
✨  Done in 3.80s.

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   yarn install --offline
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 0.33s.

and the lodash example

~/Projects/yarn-test 🐒   yarn add lodash
yarn add v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
info Direct dependencies
└─ lodash@4.17.5
info All dependencies
└─ lodash@4.17.5
✨  Done in 0.72s.

~/Projects/yarn-test 🐒   yarn cache clean
yarn cache v1.5.1
success Cleared cache.
✨  Done in 5.34s.

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   yarn install --offline
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 1.11s.

I have a Jenkins CI server building one of my projects almost daily using the offline cache (in a clean docker container that doesn't have any global yarn cache) and have never had an issue with it not resolving the packages in the offline cache.

If you look in the ./npm-packages-offline-cache dir, are the files there?


Edit:

Actually, you might be on to something here... it seems like sometimes the offline cache dir doesn't get all the package files put into it.

~/Projects/yarn-test 🐒   echo "yarn-offline-mirror \"./npm-packages-offline-cache\"" > .yarnrc

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   yarn
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 0.52s.

~/Projects/yarn-test 🐒   ls npm-packages-offline-cache/
left-pad-1.2.0.tgz

~/Projects/yarn-test 🐒   cat package.json
{
  "name": "yarn-test",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "left-pad": "^1.2.0",
    "lodash": "^4.17.5",
    "rollup": "^0.57.1"
  }
}

despite my project having 3 dependencies, only left-pad ends up in the offline cache folder, which will certainly cause problems later on during an --offline install with no cache. 🤔 Now how the heck did that happen?

~/Projects/yarn-test 🐒   yarn cache clean
yarn cache v1.5.1
success Cleared cache.
✨  Done in 0.20s.

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   yarn
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 1.44s.

~/Projects/yarn-test 🐒   ls npm-packages-offline-cache/
@types-acorn-4.0.3.tgz              for-in-1.0.2.tgz                    is-reference-1.1.0.tgz              pretty-ms-3.1.0.tgz
@types-estree-0.0.38.tgz            for-own-0.1.5.tgz                   isarray-1.0.0.tgz                   randomatic-1.1.7.tgz
acorn-5.5.3.tgz                     glob-base-0.3.0.tgz                 isobject-2.1.0.tgz                  regex-cache-0.4.4.tgz
acorn-dynamic-import-3.0.0.tgz      glob-parent-2.0.0.tgz               kind-of-3.2.2.tgz                   remove-trailing-separator-1.1.0.tgz
arr-diff-2.0.0.tgz                  irregular-plurals-1.4.0.tgz         kind-of-4.0.0.tgz                   repeat-element-1.1.2.tgz
arr-flatten-1.1.0.tgz               is-buffer-1.1.6.tgz                 left-pad-1.2.0.tgz                  repeat-string-1.6.1.tgz
array-unique-0.2.1.tgz              is-dotfile-1.0.3.tgz                locate-character-2.0.5.tgz          require-relative-0.8.7.tgz
braces-1.8.5.tgz                    is-equal-shallow-0.1.3.tgz          lodash-4.17.5.tgz                   rollup-0.57.1.tgz
date-time-2.1.0.tgz                 is-extendable-0.1.1.tgz             micromatch-2.3.11.tgz               rollup-pluginutils-2.0.1.tgz
estree-walker-0.3.1.tgz             is-extglob-1.0.0.tgz                normalize-path-2.1.1.tgz            signal-exit-3.0.2.tgz
expand-brackets-0.1.5.tgz           is-glob-2.0.1.tgz                   object.omit-2.0.1.tgz               sourcemap-codec-1.4.1.tgz
expand-range-1.8.2.tgz              is-number-2.1.0.tgz                 parse-glob-3.0.4.tgz                time-zone-1.0.0.tgz
extglob-0.3.2.tgz                   is-number-3.0.0.tgz                 parse-ms-1.0.1.tgz
filename-regex-2.0.1.tgz            is-posix-bracket-0.1.1.tgz          plur-2.1.2.tgz
fill-range-2.2.3.tgz                is-primitive-2.0.0.tgz              preserve-0.2.0.tgz

So maybe it's hitting the cache and not copying packages to the offline mirror? Except that left-pad was also in my cache but that one was copied into the offline cache the first time... very weird.


Edit 2

Now I can't reproduce the thing I just saw above. The same commands do get all the packages intot he offline cache dir:

~/Projects/yarn-test 🐒   rm -rf npm-packages-offline-cache/

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   echo "yarn-offline-mirror \"./npm-packages-offline-cache\"" > .yarnrc

~/Projects/yarn-test 🐒   yarn
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 0.52s.

~/Projects/yarn-test 🐒   ls npm-packages-offline-cache/
@types-acorn-4.0.3.tgz              fill-range-2.2.3.tgz                is-posix-bracket-0.1.1.tgz          plur-2.1.2.tgz
@types-estree-0.0.38.tgz            for-in-1.0.2.tgz                    is-primitive-2.0.0.tgz              preserve-0.2.0.tgz
acorn-5.5.3.tgz                     for-own-0.1.5.tgz                   is-reference-1.1.0.tgz              pretty-ms-3.1.0.tgz
acorn-dynamic-import-3.0.0.tgz      glob-base-0.3.0.tgz                 isarray-1.0.0.tgz                   randomatic-1.1.7.tgz
arr-diff-2.0.0.tgz                  glob-parent-2.0.0.tgz               isobject-2.1.0.tgz                  regex-cache-0.4.4.tgz
arr-flatten-1.1.0.tgz               irregular-plurals-1.4.0.tgz         kind-of-3.2.2.tgz                   remove-trailing-separator-1.1.0.tgz
array-unique-0.2.1.tgz              is-buffer-1.1.6.tgz                 kind-of-4.0.0.tgz                   repeat-element-1.1.2.tgz
braces-1.8.5.tgz                    is-dotfile-1.0.3.tgz                locate-character-2.0.5.tgz          repeat-string-1.6.1.tgz
date-time-2.1.0.tgz                 is-equal-shallow-0.1.3.tgz          lodash-4.17.5.tgz                   require-relative-0.8.7.tgz
estree-walker-0.3.1.tgz             is-extendable-0.1.1.tgz             micromatch-2.3.11.tgz               rollup-0.57.1.tgz
expand-brackets-0.1.5.tgz           is-extglob-1.0.0.tgz                normalize-path-2.1.1.tgz            rollup-pluginutils-2.0.1.tgz
expand-range-1.8.2.tgz              is-glob-2.0.1.tgz                   object.omit-2.0.1.tgz               signal-exit-3.0.2.tgz
extglob-0.3.2.tgz                   is-number-2.1.0.tgz                 parse-glob-3.0.4.tgz                sourcemap-codec-1.4.1.tgz
filename-regex-2.0.1.tgz            is-number-3.0.0.tgz                 parse-ms-1.0.1.tgz                  time-zone-1.0.0.tgz

so... I'm not sure what happened when the files didn't end up in the offline cache dir. There must be some odd edge case that causes some packages to get skipped.

Contributor

rally25rs commented Mar 19, 2018

@erickj I can't seem to reproduce what you are experiencing. (see edit below)

~/Projects/yarn-test 🐒   yarn init -y
yarn init v1.5.1
warning The yes flag has been set. This will automatically answer yes to all questions, which may have security implications.
success Saved package.json
✨  Done in 0.04s.

~/Projects/yarn-test 🐒   cat .yarnrc
yarn-offline-mirror "./npm-packages-offline-cache"
yarn-offline-mirror-pruning true

~/Projects/yarn-test 🐒   yarn add rollup
yarn add v1.5.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 48 new dependencies.
info Direct dependencies
└─ rollup@0.57.1
info All dependencies
├─ @types/acorn@4.0.3
├─ @types/estree@0.0.38
├─ acorn-dynamic-import@3.0.0
├─ acorn@5.5.3
├─ arr-diff@2.0.0
├─ arr-flatten@1.1.0
├─ array-unique@0.2.1
├─ braces@1.8.5
├─ date-time@2.1.0
├─ estree-walker@0.3.1
├─ expand-brackets@0.1.5
├─ expand-range@1.8.2
├─ extglob@0.3.2
├─ filename-regex@2.0.1
├─ fill-range@2.2.3
├─ for-in@1.0.2
├─ for-own@0.1.5
├─ glob-base@0.3.0
├─ glob-parent@2.0.0
├─ irregular-plurals@1.4.0
├─ is-dotfile@1.0.3
├─ is-equal-shallow@0.1.3
├─ is-extendable@0.1.1
├─ is-number@2.1.0
├─ is-posix-bracket@0.1.1
├─ is-primitive@2.0.0
├─ is-reference@1.1.0
├─ isarray@1.0.0
├─ isobject@2.1.0
├─ locate-character@2.0.5
├─ micromatch@2.3.11
├─ normalize-path@2.1.1
├─ object.omit@2.0.1
├─ parse-glob@3.0.4
├─ parse-ms@1.0.1
├─ plur@2.1.2
├─ preserve@0.2.0
├─ pretty-ms@3.1.0
├─ randomatic@1.1.7
├─ regex-cache@0.4.4
├─ remove-trailing-separator@1.1.0
├─ repeat-string@1.6.1
├─ require-relative@0.8.7
├─ rollup-pluginutils@2.0.1
├─ rollup@0.57.1
├─ signal-exit@3.0.2
├─ sourcemap-codec@1.4.1
└─ time-zone@1.0.0
✨  Done in 3.80s.

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   yarn install --offline
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 0.33s.

and the lodash example

~/Projects/yarn-test 🐒   yarn add lodash
yarn add v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
info Direct dependencies
└─ lodash@4.17.5
info All dependencies
└─ lodash@4.17.5
✨  Done in 0.72s.

~/Projects/yarn-test 🐒   yarn cache clean
yarn cache v1.5.1
success Cleared cache.
✨  Done in 5.34s.

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   yarn install --offline
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 1.11s.

I have a Jenkins CI server building one of my projects almost daily using the offline cache (in a clean docker container that doesn't have any global yarn cache) and have never had an issue with it not resolving the packages in the offline cache.

If you look in the ./npm-packages-offline-cache dir, are the files there?


Edit:

Actually, you might be on to something here... it seems like sometimes the offline cache dir doesn't get all the package files put into it.

~/Projects/yarn-test 🐒   echo "yarn-offline-mirror \"./npm-packages-offline-cache\"" > .yarnrc

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   yarn
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 0.52s.

~/Projects/yarn-test 🐒   ls npm-packages-offline-cache/
left-pad-1.2.0.tgz

~/Projects/yarn-test 🐒   cat package.json
{
  "name": "yarn-test",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "left-pad": "^1.2.0",
    "lodash": "^4.17.5",
    "rollup": "^0.57.1"
  }
}

despite my project having 3 dependencies, only left-pad ends up in the offline cache folder, which will certainly cause problems later on during an --offline install with no cache. 🤔 Now how the heck did that happen?

~/Projects/yarn-test 🐒   yarn cache clean
yarn cache v1.5.1
success Cleared cache.
✨  Done in 0.20s.

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   yarn
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 1.44s.

~/Projects/yarn-test 🐒   ls npm-packages-offline-cache/
@types-acorn-4.0.3.tgz              for-in-1.0.2.tgz                    is-reference-1.1.0.tgz              pretty-ms-3.1.0.tgz
@types-estree-0.0.38.tgz            for-own-0.1.5.tgz                   isarray-1.0.0.tgz                   randomatic-1.1.7.tgz
acorn-5.5.3.tgz                     glob-base-0.3.0.tgz                 isobject-2.1.0.tgz                  regex-cache-0.4.4.tgz
acorn-dynamic-import-3.0.0.tgz      glob-parent-2.0.0.tgz               kind-of-3.2.2.tgz                   remove-trailing-separator-1.1.0.tgz
arr-diff-2.0.0.tgz                  irregular-plurals-1.4.0.tgz         kind-of-4.0.0.tgz                   repeat-element-1.1.2.tgz
arr-flatten-1.1.0.tgz               is-buffer-1.1.6.tgz                 left-pad-1.2.0.tgz                  repeat-string-1.6.1.tgz
array-unique-0.2.1.tgz              is-dotfile-1.0.3.tgz                locate-character-2.0.5.tgz          require-relative-0.8.7.tgz
braces-1.8.5.tgz                    is-equal-shallow-0.1.3.tgz          lodash-4.17.5.tgz                   rollup-0.57.1.tgz
date-time-2.1.0.tgz                 is-extendable-0.1.1.tgz             micromatch-2.3.11.tgz               rollup-pluginutils-2.0.1.tgz
estree-walker-0.3.1.tgz             is-extglob-1.0.0.tgz                normalize-path-2.1.1.tgz            signal-exit-3.0.2.tgz
expand-brackets-0.1.5.tgz           is-glob-2.0.1.tgz                   object.omit-2.0.1.tgz               sourcemap-codec-1.4.1.tgz
expand-range-1.8.2.tgz              is-number-2.1.0.tgz                 parse-glob-3.0.4.tgz                time-zone-1.0.0.tgz
extglob-0.3.2.tgz                   is-number-3.0.0.tgz                 parse-ms-1.0.1.tgz
filename-regex-2.0.1.tgz            is-posix-bracket-0.1.1.tgz          plur-2.1.2.tgz
fill-range-2.2.3.tgz                is-primitive-2.0.0.tgz              preserve-0.2.0.tgz

So maybe it's hitting the cache and not copying packages to the offline mirror? Except that left-pad was also in my cache but that one was copied into the offline cache the first time... very weird.


Edit 2

Now I can't reproduce the thing I just saw above. The same commands do get all the packages intot he offline cache dir:

~/Projects/yarn-test 🐒   rm -rf npm-packages-offline-cache/

~/Projects/yarn-test 🐒   rm -rf node_modules/

~/Projects/yarn-test 🐒   echo "yarn-offline-mirror \"./npm-packages-offline-cache\"" > .yarnrc

~/Projects/yarn-test 🐒   yarn
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 0.52s.

~/Projects/yarn-test 🐒   ls npm-packages-offline-cache/
@types-acorn-4.0.3.tgz              fill-range-2.2.3.tgz                is-posix-bracket-0.1.1.tgz          plur-2.1.2.tgz
@types-estree-0.0.38.tgz            for-in-1.0.2.tgz                    is-primitive-2.0.0.tgz              preserve-0.2.0.tgz
acorn-5.5.3.tgz                     for-own-0.1.5.tgz                   is-reference-1.1.0.tgz              pretty-ms-3.1.0.tgz
acorn-dynamic-import-3.0.0.tgz      glob-base-0.3.0.tgz                 isarray-1.0.0.tgz                   randomatic-1.1.7.tgz
arr-diff-2.0.0.tgz                  glob-parent-2.0.0.tgz               isobject-2.1.0.tgz                  regex-cache-0.4.4.tgz
arr-flatten-1.1.0.tgz               irregular-plurals-1.4.0.tgz         kind-of-3.2.2.tgz                   remove-trailing-separator-1.1.0.tgz
array-unique-0.2.1.tgz              is-buffer-1.1.6.tgz                 kind-of-4.0.0.tgz                   repeat-element-1.1.2.tgz
braces-1.8.5.tgz                    is-dotfile-1.0.3.tgz                locate-character-2.0.5.tgz          repeat-string-1.6.1.tgz
date-time-2.1.0.tgz                 is-equal-shallow-0.1.3.tgz          lodash-4.17.5.tgz                   require-relative-0.8.7.tgz
estree-walker-0.3.1.tgz             is-extendable-0.1.1.tgz             micromatch-2.3.11.tgz               rollup-0.57.1.tgz
expand-brackets-0.1.5.tgz           is-extglob-1.0.0.tgz                normalize-path-2.1.1.tgz            rollup-pluginutils-2.0.1.tgz
expand-range-1.8.2.tgz              is-glob-2.0.1.tgz                   object.omit-2.0.1.tgz               signal-exit-3.0.2.tgz
extglob-0.3.2.tgz                   is-number-2.1.0.tgz                 parse-glob-3.0.4.tgz                sourcemap-codec-1.4.1.tgz
filename-regex-2.0.1.tgz            is-number-3.0.0.tgz                 parse-ms-1.0.1.tgz                  time-zone-1.0.0.tgz

so... I'm not sure what happened when the files didn't end up in the offline cache dir. There must be some odd edge case that causes some packages to get skipped.

@sbowler

This comment has been minimized.

Show comment
Hide comment
@sbowler

sbowler Aug 9, 2018

I thought I'd note this here as it seems somewhat still ongoing and my issue seems similar.

I seem to be running into a very similar odd edge case. We tried to add the package apidoc - https://github.com/apidoc/apidoc as a dev dependency and it doesn't seem to have added all of the necessary files to the offline cache, but only on Linux.

Running with NodeJs version 10 ./bin/yarn-1.7.0.js install --offline --ignore-platform on windows 10/Server 2016 everything works fine. Running the same command on Centos 7 results in errors:

error Can't make a request in offline mode ("https://registry.yarnpkg.com/semver/-/semver-5.3.0.tgz"
After manually adding that package to the offline cache, then results in
error Can't make a request in offline mode ("https://registry.yarnpkg.com/entities/-/entities-1.1.1.tgz

Maybe this is something different, but it's interesting that this is only causing problems on our Linux build. I haven't yet been able to try a newer version of yarn so I'm not sure if that would have any impact.

These also seem like dev dependencies of our dev dependencies, which I wouldn't think yarn should care about. Does yarn download all dev dependencies of every package?

(When using yarn 1.3.2 on another machine it does seem to include all missing packages so perhaps there's a bug introduced later causing this)

sbowler commented Aug 9, 2018

I thought I'd note this here as it seems somewhat still ongoing and my issue seems similar.

I seem to be running into a very similar odd edge case. We tried to add the package apidoc - https://github.com/apidoc/apidoc as a dev dependency and it doesn't seem to have added all of the necessary files to the offline cache, but only on Linux.

Running with NodeJs version 10 ./bin/yarn-1.7.0.js install --offline --ignore-platform on windows 10/Server 2016 everything works fine. Running the same command on Centos 7 results in errors:

error Can't make a request in offline mode ("https://registry.yarnpkg.com/semver/-/semver-5.3.0.tgz"
After manually adding that package to the offline cache, then results in
error Can't make a request in offline mode ("https://registry.yarnpkg.com/entities/-/entities-1.1.1.tgz

Maybe this is something different, but it's interesting that this is only causing problems on our Linux build. I haven't yet been able to try a newer version of yarn so I'm not sure if that would have any impact.

These also seem like dev dependencies of our dev dependencies, which I wouldn't think yarn should care about. Does yarn download all dev dependencies of every package?

(When using yarn 1.3.2 on another machine it does seem to include all missing packages so perhaps there's a bug introduced later causing this)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment