New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`yarn install --production` doesn't install correct dependencies #761

Closed
donovan-graham opened this Issue Oct 11, 2016 · 112 comments

Comments

Projects
None yet
@donovan-graham

donovan-graham commented Oct 11, 2016

When running yarn install --production it does not install the required dependencies of forever. This seem to be related to havingnodemon in devDependencies.

Error response:

> forever app.js
module.js:457
    throw err;
    ^
Error: Cannot find module 'minimatch'

I've created a test application here:
https://github.com/donovan-graham/yarn-example-app

#  Steps to reproduce error
git clone https://github.com/donovan-graham/yarn-example-app.git
cd yarn-example-app
yarn install --production
npm start

#  temporary step to bypass error
rm -rf node_modules
yarn remove nodemon
yarn install --production
npm start

@Daniel15 Daniel15 added the cat-bug label Oct 11, 2016

@sendilkumarn

This comment has been minimized.

Show comment
Hide comment
@sendilkumarn

sendilkumarn Oct 12, 2016

Contributor

@Daniel15 I guess this is because of nodemon is having latest version of minimatch.

The linker function currently takes in both deps and dev deps into it. For argument production, this should be prevented.

Even on normal yarn install with out production argument. Only latest version is installed in the actual path. This has to checked too.

Contributor

sendilkumarn commented Oct 12, 2016

@Daniel15 I guess this is because of nodemon is having latest version of minimatch.

The linker function currently takes in both deps and dev deps into it. For argument production, this should be prevented.

Even on normal yarn install with out production argument. Only latest version is installed in the actual path. This has to checked too.

@liorbrauer

This comment has been minimized.

Show comment
Hide comment
@liorbrauer

liorbrauer Oct 13, 2016

I'm getting a similar problem when running yarn install --production and then trying to run my build using webpack (running yarn install works fine).

> NODE_ENV=production webpack -p --config webpack/production.config.js

module.js:457
    throw err;
    ^

Error: Cannot find module 'graceful-fs'
    at Function.Module._resolveFilename (module.js:455:15)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)

and if I recall correctly, previous attempts showed similar errors with another package (not only graceful-fs)

liorbrauer commented Oct 13, 2016

I'm getting a similar problem when running yarn install --production and then trying to run my build using webpack (running yarn install works fine).

> NODE_ENV=production webpack -p --config webpack/production.config.js

module.js:457
    throw err;
    ^

Error: Cannot find module 'graceful-fs'
    at Function.Module._resolveFilename (module.js:455:15)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)

and if I recall correctly, previous attempts showed similar errors with another package (not only graceful-fs)

@comfroels

This comment has been minimized.

Show comment
Hide comment
@comfroels

comfroels Oct 14, 2016

I'm getting very similar as well... yarn install works just fine. but with the --production flag I get this:

> yarn install --production

yarn install v0.15.1
error npm-shrinkwrap.json found. This will not be updated or respected. See [TODO] for more information.
[1/4] Resolving packages...
[2/4] Fetching packages...
warning lodash@2.4.2: The engine "rhino" appears to be invalid.
warning fsevents@1.0.14: The platform "win32" is incompatible with this module.
info "fsevents@1.0.14" is an optional dependency and failed compatibility check. Excluding it from installation.
warning benchmark@1.0.0: The engine "rhino" appears to be invalid.
warning lodash@1.0.2: The engine "rhino" appears to be invalid.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[1/1] ⠐ node-sass:     at Module.require (module.js:367:17)
[-/1] ⠐ waiting...
[-/1] ⠐ waiting...
[-/1] ⠐ waiting...
error C:\vagrant\ebroker-quoteengine\node_modules\node-sass: Command failed.
Exit code: 1
Command: C:\WINDOWS\system32\cmd.exe
Arguments: /d /s /c node scripts/install.js
Directory: C:\vagrant\ebroker-quoteengine\node_modules\node-sass
Output:
module.js:341
    throw err;
    ^

Error: Cannot find module 'tough-cookie'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at Object.<anonymous> (C:\vagrant\ebroker-quoteengine\node_modules\node-sass\node_modules\request\lib\cookies.js:3:13)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Module.require (module.js:367:17)
    at SpawnError (C:\Users\nathan.white\AppData\Roaming\npm\node_modules\yarnpkg\lib\errors.js:18:1)
    at ChildProcess.<anonymous> (C:\Users\nathan.white\AppData\Roaming\npm\node_modules\yarnpkg\lib\util\child.js:107:15)
    at emitTwo (events.js:100:13)
    at ChildProcess.emit (events.js:185:7)
    at maybeClose (internal/child_process.js:827:16)
    at Socket.<anonymous> (internal/child_process.js:319:11)
    at emitOne (events.js:90:13)
    at Socket.emit (events.js:182:7)
    at Pipe._onclose (net.js:471:12)

comfroels commented Oct 14, 2016

I'm getting very similar as well... yarn install works just fine. but with the --production flag I get this:

> yarn install --production

yarn install v0.15.1
error npm-shrinkwrap.json found. This will not be updated or respected. See [TODO] for more information.
[1/4] Resolving packages...
[2/4] Fetching packages...
warning lodash@2.4.2: The engine "rhino" appears to be invalid.
warning fsevents@1.0.14: The platform "win32" is incompatible with this module.
info "fsevents@1.0.14" is an optional dependency and failed compatibility check. Excluding it from installation.
warning benchmark@1.0.0: The engine "rhino" appears to be invalid.
warning lodash@1.0.2: The engine "rhino" appears to be invalid.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[1/1] ⠐ node-sass:     at Module.require (module.js:367:17)
[-/1] ⠐ waiting...
[-/1] ⠐ waiting...
[-/1] ⠐ waiting...
error C:\vagrant\ebroker-quoteengine\node_modules\node-sass: Command failed.
Exit code: 1
Command: C:\WINDOWS\system32\cmd.exe
Arguments: /d /s /c node scripts/install.js
Directory: C:\vagrant\ebroker-quoteengine\node_modules\node-sass
Output:
module.js:341
    throw err;
    ^

Error: Cannot find module 'tough-cookie'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at Object.<anonymous> (C:\vagrant\ebroker-quoteengine\node_modules\node-sass\node_modules\request\lib\cookies.js:3:13)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Module.require (module.js:367:17)
    at SpawnError (C:\Users\nathan.white\AppData\Roaming\npm\node_modules\yarnpkg\lib\errors.js:18:1)
    at ChildProcess.<anonymous> (C:\Users\nathan.white\AppData\Roaming\npm\node_modules\yarnpkg\lib\util\child.js:107:15)
    at emitTwo (events.js:100:13)
    at ChildProcess.emit (events.js:185:7)
    at maybeClose (internal/child_process.js:827:16)
    at Socket.<anonymous> (internal/child_process.js:319:11)
    at emitOne (events.js:90:13)
    at Socket.emit (events.js:182:7)
    at Pipe._onclose (net.js:471:12)
@jkrems

This comment has been minimized.

Show comment
Hide comment
@jkrems

jkrems Oct 19, 2016

Contributor

Can reproduce a similar issue with:

npm init --yes
yarn add --dev nodemon
yarn add gulp
rm -rf node_modules
yarn install --production

This will install is-glob but not its dependency is-extglob:

> yarn why is-glob
yarn why v0.16.0
# ...
info Reasons this module exists
   - "nodemon#chokidar" depends on it
   - "gulp#liftoff#findup-sync" depends on it

> yarn why is-extglob
yarn why v0.16.0
#  ...
info This module exists because "nodemon#chokidar#is-glob" depends on it.

It seems to "forget" the gulp#liftoff dependency path while traversing..?

EDIT: Smaller example:

npm init --yes
yarn add --dev chokidar@1.6.1
yarn add liftoff@2.3.0
rm -rf node_modules
yarn --prod
node -e "require('is-glob')"

Also confirmed that removing devDependencies before running yarn --prod installs the correct dependency tree.

Contributor

jkrems commented Oct 19, 2016

Can reproduce a similar issue with:

npm init --yes
yarn add --dev nodemon
yarn add gulp
rm -rf node_modules
yarn install --production

This will install is-glob but not its dependency is-extglob:

> yarn why is-glob
yarn why v0.16.0
# ...
info Reasons this module exists
   - "nodemon#chokidar" depends on it
   - "gulp#liftoff#findup-sync" depends on it

> yarn why is-extglob
yarn why v0.16.0
#  ...
info This module exists because "nodemon#chokidar#is-glob" depends on it.

It seems to "forget" the gulp#liftoff dependency path while traversing..?

EDIT: Smaller example:

npm init --yes
yarn add --dev chokidar@1.6.1
yarn add liftoff@2.3.0
rm -rf node_modules
yarn --prod
node -e "require('is-glob')"

Also confirmed that removing devDependencies before running yarn --prod installs the correct dependency tree.

@ArkadyDR

This comment has been minimized.

Show comment
Hide comment
@ArkadyDR

ArkadyDR Oct 23, 2016

My software team ran into this issue, specifically with the package prr which is a dependency of both less and pouchdb. Many other packages were also missing from the --production build but prr was the first to cause a failure in our product. This issue has been a showstopper for us as the size of our installer would significantly increase if we included the dev packages, so we've returned to using npm.

ArkadyDR commented Oct 23, 2016

My software team ran into this issue, specifically with the package prr which is a dependency of both less and pouchdb. Many other packages were also missing from the --production build but prr was the first to cause a failure in our product. This issue has been a showstopper for us as the size of our installer would significantly increase if we included the dev packages, so we've returned to using npm.

@gihrig

This comment has been minimized.

Show comment
Hide comment
@gihrig

gihrig Oct 28, 2016

FWIW: I can work around the issue by deleting devDependencies section from package.json prior to running yarn in production.

gihrig commented Oct 28, 2016

FWIW: I can work around the issue by deleting devDependencies section from package.json prior to running yarn in production.

@royriojas

This comment has been minimized.

Show comment
Hide comment
@royriojas

royriojas Oct 31, 2016

as @gihrig said running npm prune --production can remove the devDependencies which helps to workaround this issue.

royriojas commented Oct 31, 2016

as @gihrig said running npm prune --production can remove the devDependencies which helps to workaround this issue.

@tanx

This comment has been minimized.

Show comment
Hide comment
@tanx

tanx Oct 31, 2016

as @gihrig said running npm prune --production can remove the devDependencies which helps to workaround this issue.

Yarn's main advantage over npm is a deterministic node_modules dir i.e. same in dev, CI, and production out of the box. Does running npm prune --production yield that same behavior?

My current workaround is to just install devDependencies in production as well. Disk is cheap (especially on AWS) and a deterministic install is much more important to me than disk space. So my "workaround" is to just act like yarn --production does not exist right now .

tanx commented Oct 31, 2016

as @gihrig said running npm prune --production can remove the devDependencies which helps to workaround this issue.

Yarn's main advantage over npm is a deterministic node_modules dir i.e. same in dev, CI, and production out of the box. Does running npm prune --production yield that same behavior?

My current workaround is to just install devDependencies in production as well. Disk is cheap (especially on AWS) and a deterministic install is much more important to me than disk space. So my "workaround" is to just act like yarn --production does not exist right now .

@royriojas

This comment has been minimized.

Show comment
Hide comment
@royriojas

royriojas Oct 31, 2016

@tanx npm prune --production just remove the devDependencies. And in my tests always the same modules were removed. On the other hand, yes disk space is cheap, so maybe acting as if yarn --production does not exits is a better workaround :)

royriojas commented Oct 31, 2016

@tanx npm prune --production just remove the devDependencies. And in my tests always the same modules were removed. On the other hand, yes disk space is cheap, so maybe acting as if yarn --production does not exits is a better workaround :)

@tanx

This comment has been minimized.

Show comment
Hide comment
@tanx

tanx Oct 31, 2016

@tanx npm prune --production just remove the devDependencies. And in my tests always the same modules were removed.

This is precisely the "works on my machine" mentality described in the Yarn blog post. The problem is you're letting npm change the state of node_modules without yarn's integrity check via theyarn.lock file.

tanx commented Oct 31, 2016

@tanx npm prune --production just remove the devDependencies. And in my tests always the same modules were removed.

This is precisely the "works on my machine" mentality described in the Yarn blog post. The problem is you're letting npm change the state of node_modules without yarn's integrity check via theyarn.lock file.

@kylecordes

This comment has been minimized.

Show comment
Hide comment
@kylecordes

kylecordes Oct 31, 2016

Hopefully the workarounds discussed will be rendered moot soon by an yarn update to respect dev vs production deps. In the meantime, indeed there is much to groan about with the "npm prune" post-processing hack.

kylecordes commented Oct 31, 2016

Hopefully the workarounds discussed will be rendered moot soon by an yarn update to respect dev vs production deps. In the meantime, indeed there is much to groan about with the "npm prune" post-processing hack.

@jkrems

This comment has been minimized.

Show comment
Hide comment
@jkrems

jkrems Nov 1, 2016

Contributor

The yarn why thing described above is unrelated afaict. It looks to be just a side effect of how the why code is searching for the packages.

Contributor

jkrems commented Nov 1, 2016

The yarn why thing described above is unrelated afaict. It looks to be just a side effect of how the why code is searching for the packages.

@jkrems

This comment has been minimized.

Show comment
Hide comment
@jkrems

jkrems Nov 8, 2016

Contributor

Tried to find a nice way of doing this without adding an additional pass to propagate the visibility after the graph was walked once. Not sure if it would be acceptable to just split out the visibility into a separate step..?

Contributor

jkrems commented Nov 8, 2016

Tried to find a nice way of doing this without adding an additional pass to propagate the visibility after the graph was walked once. Not sure if it would be acceptable to just split out the visibility into a separate step..?

@jkrems

This comment has been minimized.

Show comment
Hide comment
@jkrems

jkrems Nov 8, 2016

Contributor

There's some interesting edge cases as well, it's not just about properly resolving visibility:

  • A is an optional dependency of a production dependency
  • B is a non-optional dependency of a dev dependency
  • C is a non-optional dependency of both

In that case the optional flag of C depends on dev vs. prod. In dev it would be non-optional, in prod it would be optional. Just inheriting the optional flag from one of the parents (or always inheriting it from the prod parent) could lead to weirdness.

Contributor

jkrems commented Nov 8, 2016

There's some interesting edge cases as well, it's not just about properly resolving visibility:

  • A is an optional dependency of a production dependency
  • B is a non-optional dependency of a dev dependency
  • C is a non-optional dependency of both

In that case the optional flag of C depends on dev vs. prod. In dev it would be non-optional, in prod it would be optional. Just inheriting the optional flag from one of the parents (or always inheriting it from the prod parent) could lead to weirdness.

jkrems added a commit to jkrems/yarn that referenced this issue Nov 8, 2016

kittens added a commit that referenced this issue Nov 11, 2016

@SimenB

This comment has been minimized.

Show comment
Hide comment
@SimenB

SimenB Nov 16, 2016

Contributor

This is still not fixed in 0.17.2 😢

Repro: https://gist.github.com/SimenB/2b179f3b6bca73ba824e1273ea38aed3

yarn

node index.js # works

yarn --prod

node index.js # explodes

/cc @jkrems

Contributor

SimenB commented Nov 16, 2016

This is still not fixed in 0.17.2 😢

Repro: https://gist.github.com/SimenB/2b179f3b6bca73ba824e1273ea38aed3

yarn

node index.js # works

yarn --prod

node index.js # explodes

/cc @jkrems

SimenB added a commit to finn-no/node-docker that referenced this issue Nov 16, 2016

@beheh

This comment has been minimized.

Show comment
Hide comment
@beheh

beheh Nov 16, 2016

Doesn't seem fixed for me either in 0.17.2 (HearthSim/Joust#169).

beheh commented Nov 16, 2016

Doesn't seem fixed for me either in 0.17.2 (HearthSim/Joust#169).

@adamreisnz

This comment has been minimized.

Show comment
Hide comment
@adamreisnz

adamreisnz Jan 16, 2017

And this was the observation/cause of the issue: #761 (comment)

You're right, that seems likely to be the core of this problem. It seems to think that nopt is part of istanbul package, instead of google-cloud and/or babel-cli, and maybe that's why it's not installing it for production environments, because istanbul is not a prod dependency.

adamreisnz commented Jan 16, 2017

And this was the observation/cause of the issue: #761 (comment)

You're right, that seems likely to be the core of this problem. It seems to think that nopt is part of istanbul package, instead of google-cloud and/or babel-cli, and maybe that's why it's not installing it for production environments, because istanbul is not a prod dependency.

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jan 16, 2017

Member

Oh, sorry, @SimenB

yarn check --prodution --verify-tree

I'll edit my comment

Member

bestander commented Jan 16, 2017

Oh, sorry, @SimenB

yarn check --prodution --verify-tree

I'll edit my comment

@SimenB

This comment has been minimized.

Show comment
Hide comment
@SimenB

SimenB Jan 16, 2017

Contributor

Doing yarn check --verify-tree --production gives a good output (it agrees with npm ls):

yarn check v0.20.0-0
error "@finn-no/express-base#nunjucks#chokidar#anymatch" not installed
error "@finn-no/express-base#unleash-client#request#json-stringify-safe" not installed
error "@finn-no/express-base#pretty-error#renderkid#css-select#domutils#dom-serializer#entities" not installed
error Found 3 errors.

@bestander I'll email you package.json, yarn.lock and the verbose install log 😄

Contributor

SimenB commented Jan 16, 2017

Doing yarn check --verify-tree --production gives a good output (it agrees with npm ls):

yarn check v0.20.0-0
error "@finn-no/express-base#nunjucks#chokidar#anymatch" not installed
error "@finn-no/express-base#unleash-client#request#json-stringify-safe" not installed
error "@finn-no/express-base#pretty-error#renderkid#css-select#domutils#dom-serializer#entities" not installed
error Found 3 errors.

@bestander I'll email you package.json, yarn.lock and the verbose install log 😄

@adamreisnz

This comment has been minimized.

Show comment
Hide comment
@adamreisnz

adamreisnz Jan 16, 2017

@dashmug do you want me to create a new ticket for that issue? It's still a problem of incorrect dependencies being installed (albeit just for production), so I think it's related to this ticket.

adamreisnz commented Jan 16, 2017

@dashmug do you want me to create a new ticket for that issue? It's still a problem of incorrect dependencies being installed (albeit just for production), so I think it's related to this ticket.

@SimenB

This comment has been minimized.

Show comment
Hide comment
@SimenB

SimenB Jan 16, 2017

Contributor

@bestander Email sent.
While @finn-no/express-base is not publicly available, the other 3 packages in the output are, so hopefully you can reproduce with solely public packages.

Should I open a new issue?

Contributor

SimenB commented Jan 16, 2017

@bestander Email sent.
While @finn-no/express-base is not publicly available, the other 3 packages in the output are, so hopefully you can reproduce with solely public packages.

Should I open a new issue?

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jan 16, 2017

Member

@adamreisnz, could you please create a new issue then?
I can reproduce it but it is a different issue from the title one

Member

bestander commented Jan 16, 2017

@adamreisnz, could you please create a new issue then?
I can reproduce it but it is a different issue from the title one

@dashmug

This comment has been minimized.

Show comment
Hide comment
@dashmug

dashmug Jan 16, 2017

It is related. Probably the same cause. Just a different symptom. I would put that as a different issue since it's not exactly the same as what the title says.

dashmug commented Jan 16, 2017

It is related. Probably the same cause. Just a different symptom. I would put that as a different issue since it's not exactly the same as what the title says.

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jan 16, 2017

Member

@SimenB, thanks, I'll have a look

Member

bestander commented Jan 16, 2017

@SimenB, thanks, I'll have a look

@adamreisnz

This comment has been minimized.

Show comment
Hide comment
@adamreisnz

adamreisnz Jan 16, 2017

Ok will do guys.

adamreisnz commented Jan 16, 2017

Ok will do guys.

@SimenB

This comment has been minimized.

Show comment
Hide comment
@SimenB

SimenB Jan 16, 2017

Contributor

@bestander Making a package wth just those 3 in the output that fails makes it reproducible on master for me with just public deps. It's not a minimal reproduce, but still

{
  "name": "app",
  "version": "1.0.0",
  "dependencies": {
    "brakes": "^2.5.1",
    "compression": "^1.6.2",
    "envalid": "^2.4.0",
    "express": "^4.14.0",
    "object.entries": "^1.0.4",
    "prom-client": "^7.0.0",
    "response-time": "^2.3.2",
    "spaden": "^7.13.1",
    "yarn-issue-repro-package": "^1.0.0"
  },
  "devDependencies": {
    "babel-preset-es2015": "^6.18.0",
    "browserify": "^13.1.1",
    "browserify-middleware": "^7.1.0",
    "cheerio": "^0.22.0",
    "codeceptjs": "^0.4.13",
    "del-cli": "^0.2.1",
    "eslint": "^3.12.2",
    "eslint-config-finn": "^1.0.1",
    "espower-loader": "^1.0.1",
    "hashmark": "^4.1.0",
    "interfake": "^1.19.0",
    "mocha": "^3.2.0",
    "nightmare": "^2.9.0",
    "nightmare-upload": "^0.1.1",
    "nock": "^9.0.2",
    "nodemon": "^1.11.0",
    "nyc": "^10.0.0",
    "power-assert": "^1.4.1",
    "sinon": "^1.17.6",
    "supertest": "^2.0.1",
    "uglify-js": "^2.7.5",
    "uglifyify": "^3.0.4"
  }
}

package.json of yarn-issue-repro-package

{
  "name": "yarn-issue-repro-package",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "nunjucks": "^2.5.2",
    "pretty-error": "^2.0.2",
    "unleash-client": "^1.0.0-beta.7"
  }
}

Interestingly, it produces 4 errors instead of 3...

$ yarn check v0.20.0-0                                                                                                   │├─ strip-ansi@3.0.1
error "yarn-issue-repro-package#nunjucks#chokidar#anymatch" not installed                                              │└─ utila@0.4.0
error "yarn-issue-repro-package#unleash-client#request#json-stringify-safe" not installed                              │✨  Done in 2.65s.
error "yarn-issue-repro-package#nunjucks#yargs#string-width#code-point-at" not installed                               │ ~/repos/yarn-issue-repro-package  vim package.json
error "yarn-issue-repro-package#pretty-error#renderkid#css-select#domutils#dom-serializer#entities" not installed      │ ~/repos/yarn-issue-repro-package  npm publish
error Found 4 errors.
Contributor

SimenB commented Jan 16, 2017

@bestander Making a package wth just those 3 in the output that fails makes it reproducible on master for me with just public deps. It's not a minimal reproduce, but still

{
  "name": "app",
  "version": "1.0.0",
  "dependencies": {
    "brakes": "^2.5.1",
    "compression": "^1.6.2",
    "envalid": "^2.4.0",
    "express": "^4.14.0",
    "object.entries": "^1.0.4",
    "prom-client": "^7.0.0",
    "response-time": "^2.3.2",
    "spaden": "^7.13.1",
    "yarn-issue-repro-package": "^1.0.0"
  },
  "devDependencies": {
    "babel-preset-es2015": "^6.18.0",
    "browserify": "^13.1.1",
    "browserify-middleware": "^7.1.0",
    "cheerio": "^0.22.0",
    "codeceptjs": "^0.4.13",
    "del-cli": "^0.2.1",
    "eslint": "^3.12.2",
    "eslint-config-finn": "^1.0.1",
    "espower-loader": "^1.0.1",
    "hashmark": "^4.1.0",
    "interfake": "^1.19.0",
    "mocha": "^3.2.0",
    "nightmare": "^2.9.0",
    "nightmare-upload": "^0.1.1",
    "nock": "^9.0.2",
    "nodemon": "^1.11.0",
    "nyc": "^10.0.0",
    "power-assert": "^1.4.1",
    "sinon": "^1.17.6",
    "supertest": "^2.0.1",
    "uglify-js": "^2.7.5",
    "uglifyify": "^3.0.4"
  }
}

package.json of yarn-issue-repro-package

{
  "name": "yarn-issue-repro-package",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "nunjucks": "^2.5.2",
    "pretty-error": "^2.0.2",
    "unleash-client": "^1.0.0-beta.7"
  }
}

Interestingly, it produces 4 errors instead of 3...

$ yarn check v0.20.0-0                                                                                                   │├─ strip-ansi@3.0.1
error "yarn-issue-repro-package#nunjucks#chokidar#anymatch" not installed                                              │└─ utila@0.4.0
error "yarn-issue-repro-package#unleash-client#request#json-stringify-safe" not installed                              │✨  Done in 2.65s.
error "yarn-issue-repro-package#nunjucks#yargs#string-width#code-point-at" not installed                               │ ~/repos/yarn-issue-repro-package  vim package.json
error "yarn-issue-repro-package#pretty-error#renderkid#css-select#domutils#dom-serializer#entities" not installed      │ ~/repos/yarn-issue-repro-package  npm publish
error Found 4 errors.
@havenchyk

This comment has been minimized.

Show comment
Hide comment
@havenchyk

havenchyk Jan 17, 2017

I'm not sure is this the same issue or not, but here is the output (tested with 0.19.1)

havenchyk commented Jan 17, 2017

I'm not sure is this the same issue or not, but here is the output (tested with 0.19.1)

@blexrob

This comment has been minimized.

Show comment
Hide comment
@blexrob

blexrob Jan 17, 2017

Contributor

I think I have found the root cause for the installation issue.
The failure to install a package can be reproduced by the following package.json:

{ "dependencies": {
    "bcrypt": "^1.0.1",
    "gamepad": "1.4.2"
  },
  "devDependencies": {
     "istanbul": "^1.0.0-alpha.2"
  }
}

And then the commands

rm -rf node_modules yarn.lock
yarn install
rm -rf node_modules
yarn install --production
npm ls abbrev

In this configuration, abbrev is not installed.

abbrev is used by istanbul and by nopt (as can be seen from yarn why abbrev). nopt is used by istanbul and node-pre-gyp (which is used by bcrypt and gamepad).

When deduping abbrev in the package hoister, the following code is used to determine the new isIgnored function of the hoisting record:

          // switch to non ignored if earlier deduped version was ignored
          if (existing.isIgnored() && !info.isIgnored()) {
            existing.isIgnored = info.isIgnored;
          }

abbrev is one of the first hoisting records to be processed. At that moment, the existing record is istanbul#abbrev (ignored because istanbul is ignored), and the duplicate record is istanbul#nopt#abbrev, which is also ignored at the time for the same reason.

Because both records are ignored at the time, the ignore function isn't adjusted as it was supposed to - because nopt will become non-ignored in a later dedup because of the dependency by node-pre-gyp. The ignore status of both records can change at any time, so the new ignore function should mix them.

And indeed, the installation problem goes away when we replace those lines with

          // switch to non ignored if earlier deduped version was ignored
          if (existing.isIgnored()) {
            if (info.isIgnored()) {
              // both are ignored now, but any one could become non ignored later on.
              let oldIsIgnored = existing.isIgnored;
              existing.isIgnored = () => oldIsIgnored() && info.isIgnored();
            } else {
              existing.isIgnored = info.isIgnored;
            }
          }
Contributor

blexrob commented Jan 17, 2017

I think I have found the root cause for the installation issue.
The failure to install a package can be reproduced by the following package.json:

{ "dependencies": {
    "bcrypt": "^1.0.1",
    "gamepad": "1.4.2"
  },
  "devDependencies": {
     "istanbul": "^1.0.0-alpha.2"
  }
}

And then the commands

rm -rf node_modules yarn.lock
yarn install
rm -rf node_modules
yarn install --production
npm ls abbrev

In this configuration, abbrev is not installed.

abbrev is used by istanbul and by nopt (as can be seen from yarn why abbrev). nopt is used by istanbul and node-pre-gyp (which is used by bcrypt and gamepad).

When deduping abbrev in the package hoister, the following code is used to determine the new isIgnored function of the hoisting record:

          // switch to non ignored if earlier deduped version was ignored
          if (existing.isIgnored() && !info.isIgnored()) {
            existing.isIgnored = info.isIgnored;
          }

abbrev is one of the first hoisting records to be processed. At that moment, the existing record is istanbul#abbrev (ignored because istanbul is ignored), and the duplicate record is istanbul#nopt#abbrev, which is also ignored at the time for the same reason.

Because both records are ignored at the time, the ignore function isn't adjusted as it was supposed to - because nopt will become non-ignored in a later dedup because of the dependency by node-pre-gyp. The ignore status of both records can change at any time, so the new ignore function should mix them.

And indeed, the installation problem goes away when we replace those lines with

          // switch to non ignored if earlier deduped version was ignored
          if (existing.isIgnored()) {
            if (info.isIgnored()) {
              // both are ignored now, but any one could become non ignored later on.
              let oldIsIgnored = existing.isIgnored;
              existing.isIgnored = () => oldIsIgnored() && info.isIgnored();
            } else {
              existing.isIgnored = info.isIgnored;
            }
          }
@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jan 17, 2017

Member

@blexrob, great find!
Would you send a PR?
There is a disabled test for "should not lose dependencies when installing with --production" in integration.js that should be fixed now

Member

bestander commented Jan 17, 2017

@blexrob, great find!
Would you send a PR?
There is a disabled test for "should not lose dependencies when installing with --production" in integration.js that should be fixed now

@blexrob

This comment has been minimized.

Show comment
Hide comment
@blexrob

blexrob Jan 17, 2017

Contributor

@bestander, just tested it, and this fix causes a stack overflow in the test you mentioned, so it can't be applied. The following cycle pops up:

d#es5-ext -> es6-symbol#es5-ext -> es6-set#es5-ext -> es6-iterator#es5-ext -> es6-map#es5-ext -> es5-ext#es6-iterator -> es6-set#es6-iterator -> es6-weak-map#es6-iterator -> event-emitter#es5-ext -> d#es5-ext

So, the naive recursive call approach is out...

Contributor

blexrob commented Jan 17, 2017

@bestander, just tested it, and this fix causes a stack overflow in the test you mentioned, so it can't be applied. The following cycle pops up:

d#es5-ext -> es6-symbol#es5-ext -> es6-set#es5-ext -> es6-iterator#es5-ext -> es6-map#es5-ext -> es5-ext#es6-iterator -> es6-set#es6-iterator -> es6-weak-map#es6-iterator -> event-emitter#es5-ext -> d#es5-ext

So, the naive recursive call approach is out...

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jan 18, 2017

Member

Yeah, I believe it needs a bit tweaking but the idea seems correct

Member

bestander commented Jan 18, 2017

Yeah, I believe it needs a bit tweaking but the idea seems correct

@Dzenly

This comment has been minimized.

Show comment
Hide comment
@Dzenly

Dzenly Jul 13, 2017

I have such a problem with phantomjs-prebuilt module (as dependence of dependence) with yarn 0.27.5-1.
So now I do dummy yarn add phantomjs-prebuilt, before yarn install --production.

Dzenly commented Jul 13, 2017

I have such a problem with phantomjs-prebuilt module (as dependence of dependence) with yarn 0.27.5-1.
So now I do dummy yarn add phantomjs-prebuilt, before yarn install --production.

Gamerweazel pushed a commit to codechangers/pledge that referenced this issue Dec 18, 2017

@adamreisnz

This comment has been minimized.

Show comment
Hide comment
@adamreisnz

adamreisnz Jan 2, 2018

I regret to say that this still seems to be a problem in Yarn 1.3.2.
My builds on Netlify are failing when I use Yarn 1.3.2 but succeeding with Yarn 0.18.2.
The build errors with a cannot find module 'are-we-there-yet' and only with production flag.

adamreisnz commented Jan 2, 2018

I regret to say that this still seems to be a problem in Yarn 1.3.2.
My builds on Netlify are failing when I use Yarn 1.3.2 but succeeding with Yarn 0.18.2.
The build errors with a cannot find module 'are-we-there-yet' and only with production flag.

@bestander

This comment has been minimized.

Show comment
Hide comment
@bestander

bestander Jan 3, 2018

Member

@adamreisnz, this thread is too big to track all issues.
Could you raise a new one with repro script?

Member

bestander commented Jan 3, 2018

@adamreisnz, this thread is too big to track all issues.
Could you raise a new one with repro script?

@adamreisnz

This comment has been minimized.

Show comment
Hide comment
@adamreisnz

adamreisnz commented Jan 3, 2018

@bestander done, thanks.

@olastor

This comment has been minimized.

Show comment
Hide comment
@olastor

olastor Aug 24, 2018

For someone who still can't get it work and doesn't want to install jq can use

$ python -c "import json; p = json.loads(open('package.json').read()); del p['devDependencies']; open('package.json', 'w').write(json.dumps(p, indent=2));"

olastor commented Aug 24, 2018

For someone who still can't get it work and doesn't want to install jq can use

$ python -c "import json; p = json.loads(open('package.json').read()); del p['devDependencies']; open('package.json', 'w').write(json.dumps(p, indent=2));"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment