New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tarball created in Windows using "meteor build" fails to deploy to hosting environment #7401

Closed
tab00 opened this Issue Jul 13, 2016 · 26 comments

Comments

Projects
None yet
8 participants
@tab00

tab00 commented Jul 13, 2016

Deployment fails if the following 2 conditions are true:

  1. Meteor app has (cheerio OR pusher-client) AND (bufferutil OR utf-8-validate) NPM packages.
  2. Tarball created locally in Windows using meteor build and --architecture os.linux.x86_64 option

Deployment of the same app succeeds if meteor build is done in the hosting environment instead of locally e.g. using Meteor Buildpack Horse. So there could be a problem with running meteor build in Windows for a Linux hosting environment.

Also, deployment of the app succeeds via local build if either cheerio is removed or bufferutil is removed (or both).

Here are more details:

Local development platform: Windows 10
Hosting environment platform: Linux
Meteor version: 1.3.4.1. Upgrading to version 1.3.4.4 did not fix the problem.

NPM packages:
cheerio
bufferutil

Error messages in deployment script log:

npm ERR! Linux 3.19.0-33-generic
npm ERR! argv "/tmp/staged/app/.meteor/heroku_build/bin/node" "/tmp/staged/app/.meteor/heroku_build/bin/npm" "rebuild"
npm ERR! node v0.10.46
npm ERR! npm  v2.15.1
npm ERR! path /tmp/staged/app/.meteor/heroku_build/app/programs/server/npm/node_modules/cheerio/node_modules/jsdom/node_modules/request/node_modules/http-signature/node_modules/sshpk/bin\sshpk-conv
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! enoent ENOENT, chmod '/tmp/staged/app/.meteor/heroku_build/app/programs/server/npm/node_modules/cheerio/node_modules/jsdom/node_modules/request/node_modules/http-signature/node_modules/sshpk/bin\sshpk-conv'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent
npm ERR! Please include the following file with any support request:
npm ERR!     /tmp/staged/app/.meteor/heroku_build/app/programs/server/npm/npm-debug.log
npm ERR! argv "/tmp/staged/app/.meteor/heroku_build/bin/node" "/tmp/staged/app/.meteor/heroku_build/bin/npm" "install"
npm ERR! node v0.10.46
npm ERR! npm  v2.15.1
npm ERR! Exit status 34
npm ERR! Linux 3.19.0-33-generic
npm ERR! code ELIFECYCLE
npm ERR! meteor-dev-bundle@0.0.0 install: `node npm-rebuild.js`
npm ERR!
npm ERR! Failed at the meteor-dev-bundle@0.0.0 install script 'node npm-rebuild.js'.
npm ERR! This is most likely a problem with the meteor-dev-bundle package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node npm-rebuild.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs meteor-dev-bundle
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /tmp/staged/app/.meteor/heroku_build/app/programs/server/npm-debug.log
Staging failed: Buildpack compilation step failed

Error reproduction procedure:
In a Windows command prompt, cd to projects directory.
Type:

meteor create test-f83jf
cd test-f83jf
npm install --save cheerio bufferutil
meteor build "..\test-f83jf-built" --architecture os.linux.x86_64 --server-only
cd ..\test-f83jf-built

Attempt to deploy the .tar.gz file to a hosting account. I have an IBM Bluemix hosting account, so I deploy the .tar.gz using Cloud Foundry CLI and my deployment script which is mostly the same as Meteor Buildpack Horse but with the meteor build step removed:
cf push test-f83jf -b https://github.com/tab00/prebuiltdeploy -c ".meteor/heroku_build/bin/node .meteor/heroku_build/app/main.js"

@tab00

This comment has been minimized.

tab00 commented Jul 13, 2016

I've updated my post above to add pusher-client and utf-8-validate NPM packages.

Both cheerio and pusher-client has the "request" NPM package as a dependency. So maybe bufferutil and utf-8-validate conflict with packages that use "request", if deploying from a tarball.

Here is part of the deployment log when attempting to deploy via tarball an app that has pusher-client and utf-8-validate packages:

> node npm-rebuild.js
npm ERR! Linux 3.19.0-33-generic
npm ERR! argv "/tmp/staged/app/.meteor/heroku_build/bin/node" "/tmp/staged/app/.meteor/heroku_build/bin/npm" "rebuild"
npm ERR! node v0.10.46
npm ERR! npm  v2.15.1
npm ERR! path /tmp/staged/app/.meteor/heroku_build/app/programs/server/npm/node_modules/pusher-client/node_modules/request/node_modules/http-signature/node_modules/sshpk/bin\sshpk-conv
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! enoent ENOENT, chmod '/tmp/staged/app/.meteor/heroku_build/app/programs/server/npm/node_modules/pusher-client/node_modules/request/node_modules/http-signature/node_modules/sshpk/bin\sshpk-conv'
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent
npm ERR! Please include the following file with any support request:
npm ERR!     /tmp/staged/app/.meteor/heroku_build/app/programs/server/npm/npm-debug.log
npm ERR! argv "/tmp/staged/app/.meteor/heroku_build/bin/node" "/tmp/staged/app/.meteor/heroku_build/bin/npm" "install"
npm ERR! node v0.10.46
npm ERR! npm  v2.15.1
npm ERR! code ELIFECYCLE
npm ERR! meteor-dev-bundle@0.0.0 install: `node npm-rebuild.js`
npm ERR! Exit status 34
npm ERR!
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! Failed at the meteor-dev-bundle@0.0.0 install script 'node npm-rebuild.js'.
npm ERR! This is most likely a problem with the meteor-dev-bundle package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node npm-rebuild.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs meteor-dev-bundle
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!
npm ERR!     npm owner ls meteor-dev-bundle
npm ERR! Please include the following file with any support request:
npm ERR!     /tmp/staged/app/.meteor/heroku_build/app/programs/server/npm-debug.log
Staging failed: Buildpack compilation step failed
npm ERR! There is likely additional logging output above.
@benjamn

This comment has been minimized.

Member

benjamn commented Jul 13, 2016

Note that meteor build --architecture only affects Atmosphere packages that have been published for multiple platforms using meteor publish-for-arch. Atmosphere packages have a mechanism for pre-publishing binaries, but npm packages generally do not.

For npm packages with binary modules, the npm rebuild command has to succeed on the target architecture. From your logs, it looks like the sshpk package may be missing some files (e.g. bin/sshpk-conv).

Does /tmp/staged/app/.meteor/heroku_build/app/programs/server/npm/npm-debug.log have any more useful information?

@tab00

This comment has been minimized.

tab00 commented Jul 13, 2016

From your logs, it looks like the sshpk package may be missing some files (e.g. bin/sshpk-conv).

Did you notice the backslash character in 'http-signature/node_modules/sshpk/bin\sshpk-conv''?
I think that comes from the package.json file of sshpk that is contained in the tarball when I build in my Windows platform. Here is the "bin" section of that file:

  "bin": {
    "sshpk-conv": "bin\\sshpk-conv",
    "sshpk-sign": "bin\\sshpk-sign",
    "sshpk-verify": "bin\\sshpk-verify"
  },

Could this be the problem?

Does /tmp/staged/app/.meteor/heroku_build/app/programs/server/npm/npm-debug.log have any more useful information?

I can only access the files when an app is running. Since deployment fails the app doesn't run, so I cannot access the files.

@benjamn

This comment has been minimized.

Member

benjamn commented Jul 13, 2016

That \ does seem noteworthy. Maybe try editing the sshpk/package.json file before running node npm-rebuild.js on Linux, just to see if that would fix it?

@benjamn

This comment has been minimized.

Member

benjamn commented Jul 13, 2016

I can reproduce the error if I manually replace the /s in the "bin": {...} section with \\s, so I believe that's the root of the problem. Unfortunately, this seems to be an npm problem: specifically, npm rebuild doesn't (always) work when the package was originally installed on Windows.

@benjamn

This comment has been minimized.

Member

benjamn commented Jul 13, 2016

I can avoid this specific problem by running npm rebuild --no-bin-links, but I'm not sure if it will manifest itself in other ways.

@tab00

This comment has been minimized.

tab00 commented Jul 13, 2016

I replaced the double backslashes to forward slashes and that fixed the problem. I had to do it for each copy of the sshpk directory (one under cheerio, another under pusher-client).

So editing the package.json file is a workaround, but there needs to be a proper solution.

There are over 960 other package.json files in my app's built bundle, and this sshpk module was the only one that caused a problem. I've looked at some of the others and they all use a forward slash as the directory separator. So why are the backslash characters only appearing in sshpk's package.json?

@benjamn

This comment has been minimized.

Member

benjamn commented Jul 13, 2016

I think it's pretty rare for an npm package to have a "directories": { "bin": "./path/to/bin" } section, which is what results in the "bin": { ... } section in the installed package.json.

Proper-ish fix coming soon!

benjamn added a commit that referenced this issue Jul 13, 2016

@tab00

This comment has been minimized.

tab00 commented Jul 13, 2016

So was it meteor build that put the backslash characters based on the currently running platform?

Windows does actually understand / as a path separator, so I don't think there needs to be any logic to determine which slash character to use. If we just use forward slashes always I think it should be fine.

Thank you for going ahead to fix it.

@benjamn

This comment has been minimized.

Member

benjamn commented Jul 13, 2016

No, I believe it was the original npm install that created the "bin": {...} section, and npm used \s when generating additional package.json properties, because the install took place on Windows. It's possible that npm should fix this by using /s everywhere, but that's up to them.

@tab00

This comment has been minimized.

tab00 commented Jul 19, 2016

I've just updated Meteor on my Windows platform to version 1.3.5.1 and ran meteor build for my app. The resulting package.json of sshpk still contains backslash characters in the bin section:

  "bin": {
    "sshpk-conv": "bin\\sshpk-conv",
    "sshpk-sign": "bin\\sshpk-sign",
    "sshpk-verify": "bin\\sshpk-verify"
  },

Did you see something different in your own testing?

@abernix

This comment has been minimized.

Member

abernix commented Jul 19, 2016

Your old mangled node_modules might still be lying around. In your project root, can you:

  1. Delete the node_modules directory
  2. Run meteor npm install
  3. Run your Meteor project once (e.g. just meteor)
  4. Try doing the meteor build
@tab00

This comment has been minimized.

tab00 commented Jul 19, 2016

Thank you. It's working after deleting the node_modules directory and running meteor npm install prior to doing meteor build. Deployment to Linux hosting is fine now without needing to edit any package.json files.

@wearhere

This comment has been minimized.

wearhere commented Aug 18, 2016

Some of this discussion seems to suggest that the bin section of package.json is auto-generated and I can tell you that that is not the case, having added one to my own package install-files.

That section means that your package has an executable it'd like to expose to consumers—npm docs. One of the nice things about this is that these executables become available from consumers' install scripts like this.

The effect of this change appears to be that Meteor does not properly register these executables, breaking packages' install scripts. I'm currently in conversation with Galaxy support about this, but also wanted to document it here.

Is this really a general issue that would require you to always pass --no-bin-links or even just pass it on Windows, or is it possible that sshpk just has a corrupt bin section? Has this issue been seen with other packages?

@glasser

This comment has been minimized.

Member

glasser commented Aug 18, 2016

It does look like bin gets autogenerated if you use the directories.bin feature of package.json instead of directly specifying bin. So you could suggest to sshpk that they use bin instead of directories.bin (or use your own fork that does that). Not sure if they would take "I depend on the ability to use npm rebuild on another platform" as a good justification though. Or perhaps npm rebuild should be fixed to regenerate bin from directories.bin: after all, you're not allowed to provide both, so if they both exist it should be regenerated?

benjamn added a commit that referenced this issue Aug 18, 2016

Remove --no-bin-links flag from default `npm rebuild` arguments.
After recent discussion on #7401,
@glasser and I think the relatively obscure benefit of --no-bin-links is
outweighed by the problems it causes for packages whose rebuild scripts
need executable scripts installed in the node_modules/.bin/ directory.
@tab00

This comment has been minimized.

tab00 commented May 5, 2017

After creating a new project that targets the new 1.4.4.2 Meteor version I now see the exact same error return when I try to deploy to linux via tarball created in Windows:

npm ERR! enoent ENOENT: no such file or directory, chmod '/tmp/app/.meteor/heroku_build/app/programs/server/npm/node_modules/sshpk/bin\sshpk-conv'
npm ERR! enoent and is related to npm not being able to find a file.
@tab00

This comment has been minimized.

tab00 commented May 5, 2017

Editing the package.json file of sshpk as described in #7401 (comment) made deployment work as before.

Please fix this problem (again). Thank you.

@tab00

This comment has been minimized.

tab00 commented May 11, 2017

It looks like this problem has been fixed, as I now no longer need to edit the package.json file of sshpk. The relevant lines now already have forward slashes.

Thank you.

@rcurrier666

This comment has been minimized.

Contributor

rcurrier666 commented May 26, 2017

I'm seeing the problem now with Meteor 1.4.4.2. I editted the package.json in npm-modules/sshpk and that fixed the problem for now. Hopefully it won't come back, but at least there is a work-around for the problem.

@cormip

This comment has been minimized.

cormip commented Jun 19, 2017

Still a problem in Meteor 1.5. Fortunately, the previously mentioned work-around does work.

mrauhu added a commit to mrauhu/npm that referenced this issue Jul 15, 2017

Fix: normalize windows-like slashes in path, to be unix-like
When try to deploy a Meteor project from the Windows host to the Linux server
using the `mup deploy` or `meteor build --architecture os.linux.x86_64`
all binary packages broken, because NPM try to rebuild packages,
and failed on `node-sshpk` dependency of `bcrypt`.

Linked issues:

* meteor/meteor#7401
* abernix/meteord#8
* zodern/meteor-up#267
@tab00

This comment has been minimized.

tab00 commented Jul 17, 2017

I've just started using version 1.5.x and the problem has returned. So I've now had to go back to using the workaround - in package.json of sshpk change:

  "bin": {
    "sshpk-conv": "bin\\sshpk-conv",
    "sshpk-sign": "bin\\sshpk-sign",
    "sshpk-verify": "bin\\sshpk-verify"
  },

to

  "bin": {
    "sshpk-conv": "bin/sshpk-conv",
    "sshpk-sign": "bin/sshpk-sign",
    "sshpk-verify": "bin/sshpk-verify"
  },

I hope this will be fixed permanently soon.

@mrauhu

This comment has been minimized.

Contributor

mrauhu commented Jul 17, 2017

@tab00 I have two solutions for you:

  • If you are working with Meteor Up, then open mup.js and change value of field docker.image to mrauhu/meteord:base
  • For standalone Linux installation:
  1. Install patched NPM @mrauhu/npm:
npm install -g @mrauhu/npm@4.6.1 # or 5.3.0
  1. Check NPM version:
npm -v
# 4.6.1 or 5.3.0
  1. Optionally, if NPM failed set right path:
# Fix path to the scoped NPM
#
# Warning:
# /opt/nodejs is example Node.js location, change it
#
rm /opt/nodejs/bin/npm
echo 'node /opt/nodejs/lib/node_modules/@mrauhu/npm/bin/npm-cli.js "$@"' > /opt/nodejs/bin/npm
chmod +x /opt/nodejs/bin/npm

mrauhu added a commit to mrauhu/npm that referenced this issue Jul 17, 2017

Fix: normalize windows-like slashes in path, to be unix-like
When try to deploy a Meteor project from the Windows host to the Linux server
using the `mup deploy` or `meteor build --architecture os.linux.x86_64`
all binary packages broken, because NPM try to rebuild packages,
and failed on `node-sshpk` dependency of `bcrypt`.

Linked issues:

* meteor/meteor#7401
* abernix/meteord#8
* zodern/meteor-up#267
@tab00

This comment has been minimized.

tab00 commented Jul 20, 2017

@mrauhu, thanks, though I do not use Meteor Up, nor do I feel comfortable to use a patched NPM.

Will there be a proper permanent solution?

@mrauhu

This comment has been minimized.

Contributor

mrauhu commented Jul 20, 2017

@tab00, then wait for merge of the pull request in the NPM:
npm/npm#17802

@tab00

This comment has been minimized.

tab00 commented Sep 14, 2017

When can we see the fix in Meteor?

@abernix abernix added this to the Release 1.6 milestone Sep 14, 2017

@abernix

This comment has been minimized.

Member

abernix commented Sep 14, 2017

I can confirm this is fixed in Meteor 1.6, which includes the fix from npm (and more specifically, the fix applied to read-package-json in npm/read-package-json#74. Thank you very much @mrauhu for filing the issue with npm (and to the npm team for tracking down the exact fix)!

Keep in mind that Meteor 1.6 is currently a "beta", but follow that progress on #8728.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment