In some cases, meteor build doesn't include `.bin` symlinks #8136

glasser · 2016-12-06T02:04:32Z

I've seen that in some cases, meteor build doesn't include .bin symlinks from npm modules in the app directory. This can break Galaxy when it rebuilds the app, if a package that needs to be rebuilt (eg bcrypt@1.0.0) depends on a program (eg node-pre-gyp) being on the PATH.

I've actually gone and done this experiment: in an empty directory, run meteor --release 1.4.2.3 create source && cd source && meteor npm install bcrypt && meteor build --directory ../built, then tried it again including --save in the bcrypt. I saved a full diff -ruN of the directories (both source and built). (Note that in order to keep references to absolute paths in generated files constant, each "source" directory was under /tmp/experiment when I ran meteor build; I then moved them elsewhere for the diff.)

The output of the diff: https://gist.github.com/glasser/98826a5fbf5d63f0f265acf167009bd6

You can see that when I pass --save, the symlinks are missing from built/bundle/programs/server/npm/node_modules/.bin.

The only other differences are the missing bcrypt line in package.json (this is basically the definition of not passing --save), a missing _requiredBy line in bcrypt's package.json (similar), and a note in node_modules/bcrypt/build/config.gypi (in both source and built) noting that it was saved.

I'm not sure why this causes the meteor build process to drop the .bin symlinks, but it does, and this breaks the ability to run npm rebuild in the produced tarball (as Galaxy does).

The text was updated successfully, but these errors were encountered:

glasser · 2016-12-06T02:31:52Z

Note: anyone encountering this due to upgrading bcrypt specifically can work around this with meteor npm install --save bcrypt@0.8.7 — there's no new functionality in 1.0.0, just a build system change that conflicts here. (Well, there's a new Promise-based API that accounts-password does not use.)

benjamn · 2016-12-06T15:19:50Z

Adding this to the 1.4.3 milestone, but I will port the fix to previous Meteor versions as necessary.

benjamn · 2016-12-06T16:19:40Z

@glasser when you get a chance, please test this with

meteor create --release 1.4.3-beta.1 bin-links-test
cd bin-links-test
meteor npm install --save bcrypt
meteor npm install --save-dev mocha
ls -al node_modules/.bin
meteor build ../bin-links-build
cd ../bin-links-build
tar xf bin-links-test.tgz
cd bundle/programs/server
ls -al npm/node_modules/.bin
meteor npm install

The final ls -al should show a symlink for node-pre-gyp but not for mocha.

ffxsam · 2016-12-06T16:35:25Z

Hey Ben,

Thanks for checking this out! Quick Q: is the server-side warning about "Meteor is using a JavaScript implementation of bcrypt, use npm, etc" normal? In other words, will everyone see that out of the box until they run meteor npm i --save bcrypt?

benjamn · 2016-12-06T16:36:26Z

@ffxsam Only if you're depending on the Meteor bcrypt package, which probably means you're using accounts-password.

ffxsam · 2016-12-06T16:38:10Z

Yep, I am indeed. I figured that was the case.

glasser · 2016-12-06T18:11:50Z

@benjamn I verified that your steps show the symlinks, and also that the app can be successfully deployed to Galaxy without a rebuild failure.

(Note: I tend to use meteor build --directory when doing things like this!)

ffxsam · 2016-12-07T21:33:53Z

@benjamn 🍻

abernix · 2016-12-28T14:41:54Z

@vsivsi I had originally missed this issue, but I agree that this is likely the issue you are encountering that you documented in #7568 (comment).

Do you mind testing by trying 1.4.3-beta.1? It would be nice to confirm since you received a few up-votes on that comment:

meteor update --release 1.4.3-beta.1

vsivsi · 2016-12-28T20:29:31Z

@abernix I have confirmed that 1.4.3-beta.1 solves the issue I reported when deploying apps to Galaxy that require node-pre-gyp to build. Thanks for the helpful follow-up on this!

a4xrbj1 · 2017-01-02T17:01:43Z

I can confirm that the beta1 of 1.4.3 doesn't solve the problem for me!

Here's the log:

2017-01-03 00:55:55+08:00> meteor-dev-bundle@0.0.0 install /app/bundle/programs/server
v063
2017-01-03 00:55:55+08:00> node npm-rebuild.js
v063
2017-01-03 00:55:55+08:00
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00> bcrypt@1.0.2 install /app/bundle/programs/server/npm/node_modules/bcrypt
v063
2017-01-03 00:55:58+08:00> node-pre-gyp install --fallback-to-build
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00sh: 1: node-pre-gyp: not found
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00npm ERR! Linux 4.4.0-45-generic
v063
2017-01-03 00:55:58+08:00npm ERR! argv "/node-v4.6.2-linux-x64/bin/node" "/node-v4.6.2-linux-x64/bin/npm" "rebuild" "--update-binary"
v063
2017-01-03 00:55:58+08:00npm ERR! node v4.6.2
v063
2017-01-03 00:55:58+08:00npm ERR! npm v4.0.5
v063
2017-01-03 00:55:58+08:00npm ERR! file sh
v063
2017-01-03 00:55:58+08:00npm ERR! code ELIFECYCLE
v063
2017-01-03 00:55:58+08:00npm ERR! errno ENOENT
v063
2017-01-03 00:55:58+08:00npm ERR! syscall spawn
v063
2017-01-03 00:55:58+08:00npm ERR! bcrypt@1.0.2 install: `node-pre-gyp install --fallback-to-build`
v063
2017-01-03 00:55:58+08:00npm ERR! spawn ENOENT
v063
2017-01-03 00:55:58+08:00npm ERR!
v063
2017-01-03 00:55:58+08:00npm ERR! Failed at the bcrypt@1.0.2 install script 'node-pre-gyp install --fallback-to-build'.
v063
2017-01-03 00:55:58+08:00npm ERR! Make sure you have the latest version of node.js and npm installed.
v063
2017-01-03 00:55:58+08:00npm ERR! If you do, this is most likely a problem with the bcrypt package,
v063
2017-01-03 00:55:58+08:00npm ERR! not with npm itself.
v063
2017-01-03 00:55:58+08:00npm ERR! Tell the author that this fails on your system:
v063
2017-01-03 00:55:58+08:00npm ERR! node-pre-gyp install --fallback-to-build
v063
2017-01-03 00:55:58+08:00npm ERR! You can get information on how to open an issue for this project with:
v063
2017-01-03 00:55:58+08:00npm ERR! npm bugs bcrypt
v063
2017-01-03 00:55:58+08:00npm ERR! Or if that isn't available, you can get their info via:
v063
2017-01-03 00:55:58+08:00npm ERR! npm owner ls bcrypt
v063
2017-01-03 00:55:58+08:00npm ERR! There is likely additional logging output above.
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00npm ERR! Please include the following file with any support request:
v063
2017-01-03 00:55:58+08:00npm ERR! /app/bundle/programs/server/npm/npm-debug.log
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00npm WARN meteor-dev-bundle@0.0.0 No description
v063
2017-01-03 00:55:58+08:00npm WARN meteor-dev-bundle@0.0.0 No repository field.
v063
2017-01-03 00:55:58+08:00npm WARN meteor-dev-bundle@0.0.0 No license field.
v063
2017-01-03 00:55:58+08:00npm ERR! Linux 4.4.0-45-generic
v063
2017-01-03 00:55:58+08:00npm ERR! argv "/node-v4.6.2-linux-x64/bin/node" "/node-v4.6.2-linux-x64/bin/npm" "install"
v063
2017-01-03 00:55:58+08:00npm ERR! node v4.6.2
v063
2017-01-03 00:55:58+08:00npm ERR! npm v4.0.5
v063
2017-01-03 00:55:58+08:00npm ERR! code ELIFECYCLE
v063
2017-01-03 00:55:58+08:00npm ERR! meteor-dev-bundle@0.0.0 install: `node npm-rebuild.js`
v063
2017-01-03 00:55:58+08:00npm ERR! Exit status 1
v063
2017-01-03 00:55:58+08:00npm ERR!
v063
2017-01-03 00:55:58+08:00npm ERR! Failed at the meteor-dev-bundle@0.0.0 install script 'node npm-rebuild.js'.
v063
2017-01-03 00:55:58+08:00npm ERR! Make sure you have the latest version of node.js and npm installed.
v063
2017-01-03 00:55:58+08:00npm ERR! If you do, this is most likely a problem with the meteor-dev-bundle package,
v063
2017-01-03 00:55:58+08:00npm ERR! not with npm itself.
v063
2017-01-03 00:55:58+08:00npm ERR! Tell the author that this fails on your system:
v063
2017-01-03 00:55:58+08:00npm ERR! node npm-rebuild.js
v063
2017-01-03 00:55:58+08:00npm ERR! You can get information on how to open an issue for this project with:
v063
2017-01-03 00:55:58+08:00npm ERR! npm bugs meteor-dev-bundle
v063
2017-01-03 00:55:58+08:00npm ERR! Or if that isn't available, you can get their info via:
v063
2017-01-03 00:55:58+08:00npm ERR! npm owner ls meteor-dev-bundle
v063
2017-01-03 00:55:58+08:00npm ERR! There is likely additional logging output above.
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00npm ERR! Please include the following file with any support request:
v063
2017-01-03 00:55:58+08:00npm ERR! /app/bundle/programs/server/npm-debug.log
v063
2017-01-03 00:55:58+08:00Removing intermediate container 5004e5073aef

a4xrbj1 · 2017-01-02T17:22:01Z

Ok, I followed the earlier suggestion and installing the older bcrypt package solved the problem for me:

meteor npm install --save bcrypt@0.8.7

But the beta1 doesn't solve it, so more research needs to be done.

abernix · 2017-01-02T17:37:42Z

@a4xrbj1 Is this happening on a Galaxy deployment? I'd need you to break your setup again to research it further. While it's still broken, what is the output from the following:

cd your_app_directory/
ls -la node_modules/.bin

marvin-bitterlich · 2017-01-04T08:09:35Z

I got the same problem with a newer bcrypt package and the current meteor release Meteor 1.4.2.3 on galaxy (local builds fine). ls -la node_modules/.bin showed the node-pre-gyp on my local machine.

using an older version worked for me: meteor npm install --save bcrypt@0.8.7

To reproduce this: Make a new meteor app, use accounts-password with bcrypt and save the binary npm package. Ensure it has a version > 0.8.7. Then deploy to galaxy and see it crash.

chmac · 2017-01-31T18:32:10Z

In case it helps anyone else reading this, we had storybook installed which in turn depended on bcrypt 1.0.0 way down the tree. Broke galaxy deployments. :-(

ffxsam · 2017-01-31T19:11:50Z

@chmac Isn't your storybook package in the devDependencies section of your package.json file?

chmac · 2017-01-31T19:25:54Z

@ffxsam I guess it probably should have been 😜

Fixes #8136.

When `start + maxPartCount > parts.length`, this code was comparing array holes to strings, leading to unpredictable results. Blame: 2613582

mbateman · 2017-02-09T21:32:58Z

I used meteor npm install --save bcrypt@0.8.7 to solve this problem, which worked initially but has now stopped working for no apparent reason. Deployment using mupx used to be straightforward, not it's fraught with uncertainty.

glasser · 2017-02-09T21:40:55Z

Unfortunately the fix for this (released in 1.4.2.5) turned out to have some issues. 1.4.2.6 fixes them!

mbateman · 2017-02-09T21:52:42Z

Thanks for the reply. I have avoided going beyond 1.4.0.1 as I encountered open file issues and wasn't able to increase ulimit -n above 4096. Will this work in 1.4.2.6?

benjamn · 2017-02-09T22:17:30Z

@mbateman If you can't increase your ulimit, you might want to set the METEOR_WATCH_FORCE_POLLING environment variable:

export METEOR_WATCH_FORCE_POLLING=1

This increases CPU usage (because of the polling) but greatly decreases the open file count.

dr-dimitru · 2017-02-14T16:19:47Z

Hi @glasser and @benjamn ,

Looks like initial issue was fixed, when NPM dependencies of dependent packages is not loaded.
But I faced an issue when nested dependencies of dependent package is missed, but with weird behavior - modules is not found on first run and on rebuild.

Please see issue here.
What is happening:

Meteor-Files package has fs-extra as dependency
fs-extra has graceful-fs dependency which is missed on first run and on rebuild

benjamn · 2017-02-14T17:03:19Z

What version of Meteor (guessing 1.4.2.6, in which case 1.4.2.7 should fix)?
What does meteor npm version say?
Try removing the .npm directory from your package to force it to be recreated?

dr-dimitru · 2017-02-14T17:22:35Z

@benjamn thank you for quick response.

Yes meteor version was 1.4.2.6

$ meteor npm version
{ 'Meteor-Files': '1.7.9',
  npm: '4.1.2',
  ares: '1.10.1-DEV',
  http_parser: '2.7.0',
  icu: '56.1',
  modules: '46',
  node: '4.7.3',
  openssl: '1.0.2k',
  uv: '1.9.1',
  v8: '4.5.103.43',
  zlib: '1.2.8' }

After update to 1.4.2.7 (which is not solved an issue):

$ meteor npm version
{ 'Meteor-Files': '1.7.9',
  npm: '3.10.9',
  ares: '1.10.1-DEV',
  http_parser: '2.7.0',
  icu: '56.1',
  modules: '46',
  node: '4.7.3',
  openssl: '1.0.2k',
  uv: '1.9.1',
  v8: '4.5.103.43',
  zlib: '1.2.8' }

After removing .npm, run into issue and can't start application, which has ostrio:files:

$ meteor
[[[[[ ~/Sites/ostrio:files-demo/demo ]]]]]    

=> Started proxy.                             
=> Started MongoDB.                           
ostrio:files: updating npm dependencies -- fs-extra, request, throttle, file-type...
/Users/drdimitru/.meteor/packages/meteor-tool/.1.4.2_7.1krbw9w++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/lib/node_modules/meteor-promise/promise_server.js:190
      throw error;
      ^

Error: ENOENT: no such file or directory, rename '.npm/package-new-ssatz3' -> '.npm/package'
    at Error (native)
    at Object.fs.renameSync (fs.js:681:18)
    at Object.wrapper [as rename] (/tools/fs/files.js:1535:35)
    at Object.files.renameDirAlmostAtomically (/tools/fs/files.js:974:9)

dr-dimitru · 2017-02-14T17:30:15Z

@benjamn removing everything in .npm except .npm/package/npm-shrinkwrap.json and running meteor update is solved an issue.

dr-dimitru · 2017-02-14T18:22:30Z

Thank you @benjamn for quick help.
Fixed by publishing with Meteor 1.4.2.7

glasser assigned benjamn Dec 6, 2016

benjamn added this to the Release 1.4.3 milestone Dec 6, 2016

benjamn closed this as completed in f42c137 Dec 7, 2016

This was referenced Dec 28, 2016

Meteor deploy problem #8188

Closed

[1.4.0.1] node-pre-gyp needed when running npm install in bundle/programs/server #7568

Closed

benjamn added a commit that referenced this issue Feb 1, 2017

Always copy node_modules/.bin into bundle/programs/server/npm.

2613582

Fixes #8136.

benjamn added a commit that referenced this issue Feb 1, 2017

Mention #8136 in History.md for Meteor 1.4.2.4.

19b3a65

benjamn modified the milestones: Release 1.4.2.x, Release 1.4.3 Feb 1, 2017

benjamn added a commit that referenced this issue Feb 8, 2017

Fix bad npm package stripping logic introduced to fix #8136.

cafe4cc

When `start + maxPartCount > parts.length`, this code was comparing array holes to strings, leading to unpredictable results. Blame: 2613582

dr-dimitru mentioned this issue Feb 14, 2017

Cannot find module 'graceful-fs' veliovgroup/Meteor-Files#353

Closed

benjamn mentioned this issue Feb 14, 2017

[1.4.2.6] Can no longer build or deploy #8370

Closed

abernix mentioned this issue Apr 26, 2017

Error in custom deployment #8635

Closed

In some cases, meteor build doesn't include `.bin` symlinks #8136

In some cases, meteor build doesn't include `.bin` symlinks #8136

glasser commented Dec 6, 2016

glasser commented Dec 6, 2016

benjamn commented Dec 6, 2016

benjamn commented Dec 6, 2016 •

edited

ffxsam commented Dec 6, 2016

benjamn commented Dec 6, 2016 •

edited

ffxsam commented Dec 6, 2016

glasser commented Dec 6, 2016

ffxsam commented Dec 7, 2016

abernix commented Dec 28, 2016

vsivsi commented Dec 28, 2016 •

edited

a4xrbj1 commented Jan 2, 2017

a4xrbj1 commented Jan 2, 2017

abernix commented Jan 2, 2017

marvin-bitterlich commented Jan 4, 2017 •

edited

chmac commented Jan 31, 2017

ffxsam commented Jan 31, 2017

chmac commented Jan 31, 2017

mbateman commented Feb 9, 2017

glasser commented Feb 9, 2017

mbateman commented Feb 9, 2017

benjamn commented Feb 9, 2017

dr-dimitru commented Feb 14, 2017

benjamn commented Feb 14, 2017 •

edited

dr-dimitru commented Feb 14, 2017

dr-dimitru commented Feb 14, 2017 •

edited

dr-dimitru commented Feb 14, 2017

In some cases, meteor build doesn't include .bin symlinks #8136

In some cases, meteor build doesn't include .bin symlinks #8136

Comments

glasser commented Dec 6, 2016

glasser commented Dec 6, 2016

benjamn commented Dec 6, 2016

benjamn commented Dec 6, 2016 • edited

ffxsam commented Dec 6, 2016

benjamn commented Dec 6, 2016 • edited

ffxsam commented Dec 6, 2016

glasser commented Dec 6, 2016

ffxsam commented Dec 7, 2016

abernix commented Dec 28, 2016

vsivsi commented Dec 28, 2016 • edited

a4xrbj1 commented Jan 2, 2017

a4xrbj1 commented Jan 2, 2017

abernix commented Jan 2, 2017

marvin-bitterlich commented Jan 4, 2017 • edited

chmac commented Jan 31, 2017

ffxsam commented Jan 31, 2017

chmac commented Jan 31, 2017

mbateman commented Feb 9, 2017

glasser commented Feb 9, 2017

mbateman commented Feb 9, 2017

benjamn commented Feb 9, 2017

dr-dimitru commented Feb 14, 2017

benjamn commented Feb 14, 2017 • edited

dr-dimitru commented Feb 14, 2017

dr-dimitru commented Feb 14, 2017 • edited

dr-dimitru commented Feb 14, 2017

In some cases, meteor build doesn't include `.bin` symlinks #8136

In some cases, meteor build doesn't include `.bin` symlinks #8136

benjamn commented Dec 6, 2016 •

edited

benjamn commented Dec 6, 2016 •

edited

vsivsi commented Dec 28, 2016 •

edited

marvin-bitterlich commented Jan 4, 2017 •

edited

benjamn commented Feb 14, 2017 •

edited

dr-dimitru commented Feb 14, 2017 •

edited