Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In some cases, meteor build doesn't include `.bin` symlinks #8136

Closed
glasser opened this issue Dec 6, 2016 · 26 comments
Closed

In some cases, meteor build doesn't include `.bin` symlinks #8136

glasser opened this issue Dec 6, 2016 · 26 comments
Assignees

Comments

@glasser
Copy link
Member

@glasser glasser commented Dec 6, 2016

I've seen that in some cases, meteor build doesn't include .bin symlinks from npm modules in the app directory. This can break Galaxy when it rebuilds the app, if a package that needs to be rebuilt (eg bcrypt@1.0.0) depends on a program (eg node-pre-gyp) being on the PATH.

I've actually gone and done this experiment: in an empty directory, run meteor --release 1.4.2.3 create source && cd source && meteor npm install bcrypt && meteor build --directory ../built, then tried it again including --save in the bcrypt. I saved a full diff -ruN of the directories (both source and built). (Note that in order to keep references to absolute paths in generated files constant, each "source" directory was under /tmp/experiment when I ran meteor build; I then moved them elsewhere for the diff.)

The output of the diff: https://gist.github.com/glasser/98826a5fbf5d63f0f265acf167009bd6

You can see that when I pass --save, the symlinks are missing from built/bundle/programs/server/npm/node_modules/.bin.

The only other differences are the missing bcrypt line in package.json (this is basically the definition of not passing --save), a missing _requiredBy line in bcrypt's package.json (similar), and a note in node_modules/bcrypt/build/config.gypi (in both source and built) noting that it was saved.

I'm not sure why this causes the meteor build process to drop the .bin symlinks, but it does, and this breaks the ability to run npm rebuild in the produced tarball (as Galaxy does).

@glasser
Copy link
Member Author

@glasser glasser commented Dec 6, 2016

Note: anyone encountering this due to upgrading bcrypt specifically can work around this with meteor npm install --save bcrypt@0.8.7 — there's no new functionality in 1.0.0, just a build system change that conflicts here. (Well, there's a new Promise-based API that accounts-password does not use.)

@benjamn benjamn added this to the Release 1.4.3 milestone Dec 6, 2016
@benjamn
Copy link
Member

@benjamn benjamn commented Dec 6, 2016

Adding this to the 1.4.3 milestone, but I will port the fix to previous Meteor versions as necessary.

@benjamn
Copy link
Member

@benjamn benjamn commented Dec 6, 2016

@glasser when you get a chance, please test this with

meteor create --release 1.4.3-beta.1 bin-links-test
cd bin-links-test
meteor npm install --save bcrypt
meteor npm install --save-dev mocha
ls -al node_modules/.bin
meteor build ../bin-links-build
cd ../bin-links-build
tar xf bin-links-test.tgz
cd bundle/programs/server
ls -al npm/node_modules/.bin
meteor npm install

The final ls -al should show a symlink for node-pre-gyp but not for mocha.

@ffxsam
Copy link
Contributor

@ffxsam ffxsam commented Dec 6, 2016

Hey Ben,

Thanks for checking this out! Quick Q: is the server-side warning about "Meteor is using a JavaScript implementation of bcrypt, use npm, etc" normal? In other words, will everyone see that out of the box until they run meteor npm i --save bcrypt?

@benjamn
Copy link
Member

@benjamn benjamn commented Dec 6, 2016

@ffxsam Only if you're depending on the Meteor bcrypt package, which probably means you're using accounts-password.

@ffxsam
Copy link
Contributor

@ffxsam ffxsam commented Dec 6, 2016

Yep, I am indeed. I figured that was the case.

@glasser
Copy link
Member Author

@glasser glasser commented Dec 6, 2016

@benjamn I verified that your steps show the symlinks, and also that the app can be successfully deployed to Galaxy without a rebuild failure.

(Note: I tend to use meteor build --directory when doing things like this!)

@benjamn benjamn closed this in f42c137 Dec 7, 2016
@ffxsam
Copy link
Contributor

@ffxsam ffxsam commented Dec 7, 2016

@benjamn 🍻

@abernix
Copy link
Member

@abernix abernix commented Dec 28, 2016

@vsivsi I had originally missed this issue, but I agree that this is likely the issue you are encountering that you documented in #7568 (comment).

Do you mind testing by trying 1.4.3-beta.1? It would be nice to confirm since you received a few up-votes on that comment:

meteor update --release 1.4.3-beta.1
@vsivsi
Copy link

@vsivsi vsivsi commented Dec 28, 2016

@abernix I have confirmed that 1.4.3-beta.1 solves the issue I reported when deploying apps to Galaxy that require node-pre-gyp to build. Thanks for the helpful follow-up on this!

@a4xrbj1
Copy link

@a4xrbj1 a4xrbj1 commented Jan 2, 2017

I can confirm that the beta1 of 1.4.3 doesn't solve the problem for me!

Here's the log:

2017-01-03 00:55:55+08:00> meteor-dev-bundle@0.0.0 install /app/bundle/programs/server
v063
2017-01-03 00:55:55+08:00> node npm-rebuild.js
v063
2017-01-03 00:55:55+08:00
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00> bcrypt@1.0.2 install /app/bundle/programs/server/npm/node_modules/bcrypt
v063
2017-01-03 00:55:58+08:00> node-pre-gyp install --fallback-to-build
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00sh: 1: node-pre-gyp: not found
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00npm ERR! Linux 4.4.0-45-generic
v063
2017-01-03 00:55:58+08:00npm ERR! argv "/node-v4.6.2-linux-x64/bin/node" "/node-v4.6.2-linux-x64/bin/npm" "rebuild" "--update-binary"
v063
2017-01-03 00:55:58+08:00npm ERR! node v4.6.2
v063
2017-01-03 00:55:58+08:00npm ERR! npm v4.0.5
v063
2017-01-03 00:55:58+08:00npm ERR! file sh
v063
2017-01-03 00:55:58+08:00npm ERR! code ELIFECYCLE
v063
2017-01-03 00:55:58+08:00npm ERR! errno ENOENT
v063
2017-01-03 00:55:58+08:00npm ERR! syscall spawn
v063
2017-01-03 00:55:58+08:00npm ERR! bcrypt@1.0.2 install: `node-pre-gyp install --fallback-to-build`
v063
2017-01-03 00:55:58+08:00npm ERR! spawn ENOENT
v063
2017-01-03 00:55:58+08:00npm ERR!
v063
2017-01-03 00:55:58+08:00npm ERR! Failed at the bcrypt@1.0.2 install script 'node-pre-gyp install --fallback-to-build'.
v063
2017-01-03 00:55:58+08:00npm ERR! Make sure you have the latest version of node.js and npm installed.
v063
2017-01-03 00:55:58+08:00npm ERR! If you do, this is most likely a problem with the bcrypt package,
v063
2017-01-03 00:55:58+08:00npm ERR! not with npm itself.
v063
2017-01-03 00:55:58+08:00npm ERR! Tell the author that this fails on your system:
v063
2017-01-03 00:55:58+08:00npm ERR! node-pre-gyp install --fallback-to-build
v063
2017-01-03 00:55:58+08:00npm ERR! You can get information on how to open an issue for this project with:
v063
2017-01-03 00:55:58+08:00npm ERR! npm bugs bcrypt
v063
2017-01-03 00:55:58+08:00npm ERR! Or if that isn't available, you can get their info via:
v063
2017-01-03 00:55:58+08:00npm ERR! npm owner ls bcrypt
v063
2017-01-03 00:55:58+08:00npm ERR! There is likely additional logging output above.
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00npm ERR! Please include the following file with any support request:
v063
2017-01-03 00:55:58+08:00npm ERR! /app/bundle/programs/server/npm/npm-debug.log
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00npm WARN meteor-dev-bundle@0.0.0 No description
v063
2017-01-03 00:55:58+08:00npm WARN meteor-dev-bundle@0.0.0 No repository field.
v063
2017-01-03 00:55:58+08:00npm WARN meteor-dev-bundle@0.0.0 No license field.
v063
2017-01-03 00:55:58+08:00npm ERR! Linux 4.4.0-45-generic
v063
2017-01-03 00:55:58+08:00npm ERR! argv "/node-v4.6.2-linux-x64/bin/node" "/node-v4.6.2-linux-x64/bin/npm" "install"
v063
2017-01-03 00:55:58+08:00npm ERR! node v4.6.2
v063
2017-01-03 00:55:58+08:00npm ERR! npm v4.0.5
v063
2017-01-03 00:55:58+08:00npm ERR! code ELIFECYCLE
v063
2017-01-03 00:55:58+08:00npm ERR! meteor-dev-bundle@0.0.0 install: `node npm-rebuild.js`
v063
2017-01-03 00:55:58+08:00npm ERR! Exit status 1
v063
2017-01-03 00:55:58+08:00npm ERR!
v063
2017-01-03 00:55:58+08:00npm ERR! Failed at the meteor-dev-bundle@0.0.0 install script 'node npm-rebuild.js'.
v063
2017-01-03 00:55:58+08:00npm ERR! Make sure you have the latest version of node.js and npm installed.
v063
2017-01-03 00:55:58+08:00npm ERR! If you do, this is most likely a problem with the meteor-dev-bundle package,
v063
2017-01-03 00:55:58+08:00npm ERR! not with npm itself.
v063
2017-01-03 00:55:58+08:00npm ERR! Tell the author that this fails on your system:
v063
2017-01-03 00:55:58+08:00npm ERR! node npm-rebuild.js
v063
2017-01-03 00:55:58+08:00npm ERR! You can get information on how to open an issue for this project with:
v063
2017-01-03 00:55:58+08:00npm ERR! npm bugs meteor-dev-bundle
v063
2017-01-03 00:55:58+08:00npm ERR! Or if that isn't available, you can get their info via:
v063
2017-01-03 00:55:58+08:00npm ERR! npm owner ls meteor-dev-bundle
v063
2017-01-03 00:55:58+08:00npm ERR! There is likely additional logging output above.
v063
2017-01-03 00:55:58+08:00
v063
2017-01-03 00:55:58+08:00npm ERR! Please include the following file with any support request:
v063
2017-01-03 00:55:58+08:00npm ERR! /app/bundle/programs/server/npm-debug.log
v063
2017-01-03 00:55:58+08:00Removing intermediate container 5004e5073aef
@a4xrbj1
Copy link

@a4xrbj1 a4xrbj1 commented Jan 2, 2017

Ok, I followed the earlier suggestion and installing the older bcrypt package solved the problem for me:

meteor npm install --save bcrypt@0.8.7

But the beta1 doesn't solve it, so more research needs to be done.

@abernix
Copy link
Member

@abernix abernix commented Jan 2, 2017

@a4xrbj1 Is this happening on a Galaxy deployment? I'd need you to break your setup again to research it further. While it's still broken, what is the output from the following:

cd your_app_directory/
ls -la node_modules/.bin
@reggaemuffin
Copy link

@reggaemuffin reggaemuffin commented Jan 4, 2017

I got the same problem with a newer bcrypt package and the current meteor release Meteor 1.4.2.3 on galaxy (local builds fine). ls -la node_modules/.bin showed the node-pre-gyp on my local machine.

using an older version worked for me: meteor npm install --save bcrypt@0.8.7

To reproduce this: Make a new meteor app, use accounts-password with bcrypt and save the binary npm package. Ensure it has a version > 0.8.7. Then deploy to galaxy and see it crash.

@chmac
Copy link

@chmac chmac commented Jan 31, 2017

In case it helps anyone else reading this, we had storybook installed which in turn depended on bcrypt 1.0.0 way down the tree. Broke galaxy deployments. :-(

@ffxsam
Copy link
Contributor

@ffxsam ffxsam commented Jan 31, 2017

@chmac Isn't your storybook package in the devDependencies section of your package.json file?

@chmac
Copy link

@chmac chmac commented Jan 31, 2017

@ffxsam I guess it probably should have been 😜

@benjamn benjamn modified the milestones: Release 1.4.2.x, Release 1.4.3 Feb 1, 2017
benjamn added a commit that referenced this issue Feb 8, 2017
When `start + maxPartCount > parts.length`, this code was comparing
array holes to strings, leading to unpredictable results.

Blame: 2613582
@mbateman
Copy link

@mbateman mbateman commented Feb 9, 2017

I used meteor npm install --save bcrypt@0.8.7 to solve this problem, which worked initially but has now stopped working for no apparent reason. Deployment using mupx used to be straightforward, not it's fraught with uncertainty.

@glasser
Copy link
Member Author

@glasser glasser commented Feb 9, 2017

Unfortunately the fix for this (released in 1.4.2.5) turned out to have some issues. 1.4.2.6 fixes them!

@mbateman
Copy link

@mbateman mbateman commented Feb 9, 2017

Thanks for the reply. I have avoided going beyond 1.4.0.1 as I encountered open file issues and wasn't able to increase ulimit -n above 4096. Will this work in 1.4.2.6?

@benjamn
Copy link
Member

@benjamn benjamn commented Feb 9, 2017

@mbateman If you can't increase your ulimit, you might want to set the METEOR_WATCH_FORCE_POLLING environment variable:

export METEOR_WATCH_FORCE_POLLING=1

This increases CPU usage (because of the polling) but greatly decreases the open file count.

@dr-dimitru
Copy link
Contributor

@dr-dimitru dr-dimitru commented Feb 14, 2017

Hi @glasser and @benjamn ,

Looks like initial issue was fixed, when NPM dependencies of dependent packages is not loaded.
But I faced an issue when nested dependencies of dependent package is missed, but with weird behavior - modules is not found on first run and on rebuild.

Please see issue here.
What is happening:

  • Meteor-Files package has fs-extra as dependency
  • fs-extra has graceful-fs dependency which is missed on first run and on rebuild
@benjamn
Copy link
Member

@benjamn benjamn commented Feb 14, 2017

  1. What version of Meteor (guessing 1.4.2.6, in which case 1.4.2.7 should fix)?
  2. What does meteor npm version say?
  3. Try removing the .npm directory from your package to force it to be recreated?
@dr-dimitru
Copy link
Contributor

@dr-dimitru dr-dimitru commented Feb 14, 2017

@benjamn thank you for quick response.

Yes meteor version was 1.4.2.6

$ meteor npm version
{ 'Meteor-Files': '1.7.9',
  npm: '4.1.2',
  ares: '1.10.1-DEV',
  http_parser: '2.7.0',
  icu: '56.1',
  modules: '46',
  node: '4.7.3',
  openssl: '1.0.2k',
  uv: '1.9.1',
  v8: '4.5.103.43',
  zlib: '1.2.8' }

After update to 1.4.2.7 (which is not solved an issue):

$ meteor npm version
{ 'Meteor-Files': '1.7.9',
  npm: '3.10.9',
  ares: '1.10.1-DEV',
  http_parser: '2.7.0',
  icu: '56.1',
  modules: '46',
  node: '4.7.3',
  openssl: '1.0.2k',
  uv: '1.9.1',
  v8: '4.5.103.43',
  zlib: '1.2.8' }

After removing .npm, run into issue and can't start application, which has ostrio:files:

$ meteor
[[[[[ ~/Sites/ostrio:files-demo/demo ]]]]]    

=> Started proxy.                             
=> Started MongoDB.                           
ostrio:files: updating npm dependencies -- fs-extra, request, throttle, file-type...
/Users/drdimitru/.meteor/packages/meteor-tool/.1.4.2_7.1krbw9w++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/lib/node_modules/meteor-promise/promise_server.js:190
      throw error;
      ^

Error: ENOENT: no such file or directory, rename '.npm/package-new-ssatz3' -> '.npm/package'
    at Error (native)
    at Object.fs.renameSync (fs.js:681:18)
    at Object.wrapper [as rename] (/tools/fs/files.js:1535:35)
    at Object.files.renameDirAlmostAtomically (/tools/fs/files.js:974:9)
@dr-dimitru
Copy link
Contributor

@dr-dimitru dr-dimitru commented Feb 14, 2017

@benjamn removing everything in .npm except .npm/package/npm-shrinkwrap.json and running meteor update is solved an issue.

@dr-dimitru
Copy link
Contributor

@dr-dimitru dr-dimitru commented Feb 14, 2017

Thank you @benjamn for quick help.
Fixed by publishing with Meteor 1.4.2.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.