Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gRPC sporadically doesn't build properly under node 11. #594

Closed
lafama opened this issue Oct 25, 2018 · 45 comments
Closed

gRPC sporadically doesn't build properly under node 11. #594

lafama opened this issue Oct 25, 2018 · 45 comments

Comments

@lafama
Copy link

lafama commented Oct 25, 2018

Problem description

A clear and concise description of what the problem is.
After upgrading to node 11.0.0 npm install fails due to @google-cloud/language dependency which has a dependency to google-gax which has a dependency to grpc@1.15.1

 node-pre-gyp install --fallback-to-build --library=static_library

node-pre-gyp WARN Using request for node-pre-gyp https download 
node-pre-gyp WARN Tried to download(404): https://node-precompiled-binaries.grpc.io/grpc/v1.15.1/node-v67-darwin-x64-unknown.tar.gz 
node-pre-gyp WARN Pre-built binaries not found for grpc@1.15.1 and node@11.0.0 (node-v67 ABI, unknown) (falling back to source compile with node-gyp) 
node-pre-gyp WARN Pre-built binaries not installable for grpc@1.15.1 and node@11.0.0 (node-v67 ABI, unknown) (falling back to source compile with node-gyp) 
node-pre-gyp WARN Hit error Connection closed while downloading tarball file 

grpc/src/node/extension_binary/node-v67-darwin-x64-unknown --napi_version=3 --node_abi_napi=napi' (1)
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.oexec install script
../deps/grpc/src/core/lib/backoff/backoff.cc:19:10: fatal error: 'grpc/support/port_platform.h' file not found
#include <grpc/support/port_platform.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.░⸩ ⠸ install: info lifecycle grpc@1.15.1~install: Failed to exec install script
make: *** [Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.o] Error 1

Reproduction steps

Give very precise steps you've discovered to reproduce your problem. If possible and applicable, provide us with a repository we can clone that contains a reproduction case. Also if possible and applicable, please include a Dockerfile that exhibits the problem if it's specific to a certain environment. Bug reports with no reproduction steps will be closed.
Npm install
-Node version 11.0.0
-Add reference to @google-cloud/language

Environment

  • OS name, version and architecture: [e.g. Linux Ubuntu 18.04 amd64] Mac OS
  • Node version [e.g. 8.10.0] 11.0.0
  • Node installation method [e.g. nvm]
  • If applicable, compiler version [e.g. clang 3.8.0-2ubuntu4]
  • Package name and version [e.g. gRPC@1.12.0] 1.15.1

Additional context

Add any other context about the problem here. If possible, attach full logs. Do not try to omit anything for brevity, but instead include absolutely everything. Please try and set your operating system's locale to English, so that logs contain error messages in the English language.

@nicolasnoble
Copy link
Member

This looks like we have a broken source distribution for 1.15.1...

@murgatroid99
Copy link
Member

I don't know what's going on with that source distribution. I tried it myself and it works, and that file is present and in the include paths.

@nicolasnoble
Copy link
Member

Maybe it's a Node 11 thing? Did you try with this runtime?

@nicolasnoble
Copy link
Member

So I just tried installing @google-cloud/language on my macbook with node 11 installed from nvm, and things worked properly fallbacking on compiling from source, so we're not able to reproduce this one.

@nicolasnoble
Copy link
Member

From grpc/grpc#17011 - there IS something going on, it's sporadic, and horribly feels like a race condition. I don't know where the race is yet, but it's definitely related to node 11, and not with grpc itself.

@nicolasnoble
Copy link
Member

By the looks of it, there'll be a node 11.0.1 release soon, with issues in there that may be related to this. Please wait until this is released before attempting to use node 11.

@nicolasnoble nicolasnoble changed the title Npm install fails after update to node 11.0.0 due to @google-cloud/language dependency gRPC sporadically doesn't build properly under node 11. Oct 26, 2018
@jkwlui
Copy link

jkwlui commented Oct 26, 2018

Thanks for investigating, we'll wait and see if things change with node 11.0.1 :)

@ojordann
Copy link

Works fine with Node 10.12.0

@nodeful
Copy link

nodeful commented Nov 1, 2018

Running Node v10.13.0 today and GRPC fails to install using npm i -S grpc
Error:

node-pre-gyp ERR! Tried to download(403): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.1/node-v64-linux-x64-musl.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for grpc@1.9.1 and node@10.13.0 (node-v64 ABI, musl) (falling back to source compile with node-gyp) 
node-pre-gyp ERR! Tried to download(undefined): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.1/node-v64-linux-x64-musl.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for grpc@1.9.1 and node@10.13.0 (node-v64 ABI, musl) (falling back to source compile with node-gyp) 
make: Entering directory '/var/www/app/current/node_modules/grpc/build'
make: Entering directory '/var/www/app/current/node_modules/grpc/build'
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o
rm: can't remove './Release/.deps/Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o.d.raw': No such file or directory
make: *** [grpc.target.mk:394: Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o] Error 1
make: Leaving directory '/var/www/app/current/node_modules/grpc/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Linux 4.14.48-coreos-r2
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--library=static_library" "--module=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl/grpc_node.node" "--module_name=grpc_node" "--module_path=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl"
gyp ERR! cwd /var/www/app/current/node_modules/grpc
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.o
gyp ERR! node -v v10.13.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
node-pre-gyp ERR! build error 
node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --library=static_library --module=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl/grpc_node.node --module_name=grpc_node --module_path=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/var/www/app/current/node_modules/grpc/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
node-pre-gyp ERR! System Linux 4.14.48-coreos-r2
node-pre-gyp ERR! command "/usr/bin/node" "/var/www/app/current/node_modules/grpc/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build" "--library=static_library"
node-pre-gyp ERR! cwd /var/www/app/current/node_modules/grpc
node-pre-gyp ERR! node -v v10.13.0
node-pre-gyp ERR! node-pre-gyp -v v0.6.39
node-pre-gyp ERR! not ok 
Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --library=static_library --module=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl/grpc_node.node --module_name=grpc_node --module_path=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl' (1)
Assembler messages:
Error: can't open /var/www/app/current/node_modules/grpc/ccDekbog.s for reading: No such file or directory
make: *** [grpc.target.mk:394: Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.o] Error 1
make: Leaving directory '/var/www/app/current/node_modules/grpc/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Linux 4.14.48-coreos-r2
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--library=static_library" "--module=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl/grpc_node.node" "--module_name=grpc_node" "--module_path=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl"
gyp ERR! cwd /var/www/app/current/node_modules/grpc
gyp ERR! node -v v10.13.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! grpc@1.9.1 install: `node-pre-gyp install --fallback-to-build --library=static_library`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the grpc@1.9.1 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Seems like this file:
https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.1/node-v64-linux-x64-musl.tar.gz is not Publicly openned

@jggc
Copy link

jggc commented Nov 1, 2018

ArchLinux nodejs 11.0 fails EVERY time.

But with node 10.13 from nvm it works fine.

Here is my failing build log

@84564221
Copy link

84564221 commented Nov 1, 2018

gyp ERR! configure error 
gyp ERR! stack Error: Command failed: /usr/bin/python -c import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack   File "<string>", line 1
gyp ERR! stack     import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack                                ^
gyp ERR! stack SyntaxError: invalid syntax
gyp ERR! stack 
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:289:12)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)

...

gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "build" 
  "--fallback-to-build" 
  "--library=static_library" 
  "--module=/home/user/grpc-test/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc/grpc_node.node" 
  "--module_name=grpc_node" 
  "--module_path=/home/user/grpc-test/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc" 
  "--napi_version=3" 
  "--node_abi_napi=napi"
gyp ERR! cwd /home/user/grpc-test/node_modules/grpc
gyp ERR! node -v v11.0.0
gyp ERR! node-gyp -v v3.8.0

Let's have a look

$ python --version
Python 2.7.15

Same issue when switching to python3

$ python --version
Python 3.7.1

@murgatroid99
Copy link
Member

The original issue is about sporadic compilation failures under Node 11. If you are experiencing different errors under a different version of Node, please file a separate issue.

@hilongjw
Copy link

hilongjw commented Nov 2, 2018

same issue here

node-pre-gyp ERR! node -v v11.0.0
node-pre-gyp ERR! node-pre-gyp -v v0.10.3
node-pre-gyp ERR! not ok
Failed to execute '/Users/awe/.nvm/versions/node/v11.0.0/bin/node /Users/awe/.nvm/versions/node/v11.0.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --library=static_library --module=/Users/awe/Desktop/code/project/era-cloud-proxy/node_modules/grpc/src/node/extension_binary/node-v67-darwin-x64-unknown/grpc_node.node --module_name=grpc_node --module_path=/Users/awe/Desktop/code/project/era-cloud-proxy/node_modules/grpc/src/node/extension_binary/node-v67-darwin-x64-unknown --napi_version=3 --node_abi_napi=napi' (1)
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.oexec install script
../deps/grpc/src/core/lib/backoff/backoff.cc:19:10: fatal error: 'grpc/support/port_platform.h' file not
      found
#include <grpc/support/port_platform.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.░⸩ ⠹ install: info lifecycle grpc@1.16.0~install: Failed to exec install script
make: *** [Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.o] Error 1

@christopher-johnson
Copy link

reproduced using docker node:latest (11.1.0) and travis here. grpc compiles 2 times successfully, then on the third time it fails. The source binaries are not available yet as indicated with "Pre-built binaries not found for grpc@1.13.1 and node@11.1.0"

@nicolasnoble
Copy link
Member

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

@smtapp-smtio-cloud
Copy link

@nicolasnoble dude, you're a genius. You saved me from all the frustrations that I was building up due to this issue for the past week! Way to go!

@itaied246
Copy link

@nicolasnoble what about yarn?

@nicolasnoble
Copy link
Member

We've never officially supported yarn as a distribution system.

@malkhalaf
Copy link

I have the same problem running MacOS High Sierra using homebrew to install node and the installed version is 11.0. Build from sources option did not work.
Solution: I force downgraded node to v10 and install went fine.

@jscti
Copy link

jscti commented Nov 5, 2018

node 10.13 worked. Thanks @nicolasnoble

@nicolasnoble
Copy link
Member

@Djiit electron-builder is a different issue altogether. Please read the relevant section of our readme about using grpc with electron: https://github.com/grpc/grpc-node/tree/master/packages/grpc-native-core#about-electron

@swoodivyesh
Copy link

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

would you please explain me what does npm install --build-from-source will do? I mean what is the use case of --build-from-source is for npm install in general?

@swoodivyesh
Copy link

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

why only this fix works, what is the actual issue happening? is it because whenever it goes to download the http download it gets the close connection error?

@Tsangares
Copy link

Tsangares commented Nov 7, 2018

I used npm install --build-from-source @grpc/grpc-js and or npm install --build-from-source grpc both fail when building at:

CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/channel.o
../deps/grpc/src/core/lib/surface/channel.cc: In function ‘grpc_channel* grpc_channel_create_with_builder(grpc_channel_stack_builder*, grpc_channel_stack_type)’:
../deps/grpc/src/core/lib/surface/channel.cc:103:38: error: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘grpc_channel’ {aka ‘struct grpc_channel’} with no trivial copy-assignment; use assignment or value-initialization instead [-Werror=class-memaccess]
   memset(channel, 0, sizeof(*channel));
                                      ^
../deps/grpc/src/core/lib/surface/channel.cc:61:8: note: ‘grpc_channel’ {aka ‘struct grpc_channel’} declared here
 struct grpc_channel {
        ^~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make: *** [grpc.target.mk:469: Release/obj.target/grpc/deps/grpc/src/core/lib/surface/channel.o] Error 1
make: Leaving directory ./node_modules/grpc/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:246:12)
gyp ERR! System Linux 4.18.16-arch1-1-ARCH
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--library=static_library" "--module=./node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc/grpc_node.node" "--module_name=grpc_node" "--module_path=./node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc" "--napi_version=3" "--node_abi_napi=napi"
gyp ERR! cwd ./node_modules/grpc
gyp ERR! node -v v11.1.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 

Note ArchLinux OS

@nicolasnoble
Copy link
Member

@Tsangares that may be a tangential issue, where we don't properly support your compiler. It's in fact the same as grpc/grpc#15576 - is your compiler gcc 8? If yes, please follow along on the other issue.

@nicolasnoble
Copy link
Member

@swoodivyesh the explanations are on the various bugs I've filed against node-pre-gyp, and the gist of it is that there's a bug in this software that will trigger a race condition if we haven't published binaries for the platform you're trying to use grpc on. And we can't publish node 11 binaries because node-pre-gyp doesn't support it yet.

Using --build-from-source calls into another codepath of node-pre-gyp, and that avoids the race condition. Call it an ugly workaround until their bugs are fixed.

@spencerfontein
Copy link

spencerfontein commented Nov 8, 2018

Using

node 10.13.0
npm 6.4.1
OS: Ubuntu 18.04

And still getting error anyone else? I tried npm install --build-from-source but that also doesnt work

@nicolasnoble
Copy link
Member

@spencerfontein please create a new issue with the full report of the failure you are experiencing.

@swoodivyesh
Copy link

swoodivyesh commented Nov 8, 2018

@swoodivyesh the explanations are on the various bugs I've filed against node-pre-gyp, and the gist of it is that there's a bug in this software that will trigger a race condition if we haven't published binaries for the platform you're trying to use grpc on. And we can't publish node 11 binaries because node-pre-gyp doesn't support it yet.

Using --build-from-source calls into another codepath of node-pre-gyp, and that avoids the race condition. Call it an ugly workaround until their bugs are fixed.

okay so was it due to mixed content error? What I understood from it is that few libraries have resources coming in with http and others with https so is that the main issue here?

@nicolasnoble

@DeividVeloso
Copy link

I fix this using a Node version v10.13.0 after I delete node_modules and install with npm install again and works for me.

@mantielero
Copy link

It fails for me when I try to build from source in node v11:

[...]
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/ext/transport/chttp2/transport/chttp2_plugin.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/ext/transport/chttp2/transport/flow_control.o
../deps/grpc/src/core/ext/transport/chttp2/transport/flow_control.cc: In member function 'virtual uint32_t grpc_core::chttp2::TransportFlowControl::MaybeSendUpdate(bool)':
../deps/grpc/src/core/ext/transport/chttp2/transport/flow_control.cc:193:50: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
       static_cast<const uint32_t>(target_window());
                                                  ^
cc1plus: all warnings being treated as errors
make: *** [grpc.target.mk:470: Release/obj.target/grpc/deps/grpc/src/core/ext/transport/chttp2/transport/flow_control.o] Error 1
[...]

Is it my bad?

@nicolasnoble
Copy link
Member

nicolasnoble commented Nov 9, 2018

Which is a duplicate of grpc/grpc#15576

Please stop trying to use grpc with node 11 until node-pre-gyp has node 11 support. Then we'll be able to properly assist.

@mantielero
Copy link

Sorry mate.

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

So this is not applicable then.

@swoodivyesh
Copy link

Sorry mate.

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

So this is not applicable then.

it is! @mantielero that's what resolved it for me. What I understood from this problem that node-pre-gyp has few resources who fetch in data from http requests and others from https request which causes the Mixed Content Erorr you can read more about it here : https://developers.google.com/web/fundamentals/security/prevent-mixed-content/what-is-mixed-content

This is totally my understanding so correct me if I'm wrong.

@nicolasnoble
Copy link
Member

@mantielero your issue is that you can't compile from source because of grpc/grpc#15576 since your compiler isn't agreeing with the code we publish. Once we can have node-pre-gyp's node 11 support, we'll be able to publish node 11 prebuilt binaries and you won't have to compile from source.

@JesusAlbornoz
Copy link

I have to downgrade to v10.13.0 of Node, delete my node_modules folder and do npm install... Works for me

@stupidly-logical
Copy link

The only solution I found was to downgrade Node to the current lts : 10.13.0

Steps:

  1. sudo npm cache clean -f
  2. sudo npm install -g n
  3. sudo n 10.13.0
  4. npm install firebase

@nicolasnoble
Copy link
Member

I've published grpc 1.16.1, that now depends on node-pre-gyp 0.12.0. Through this update, this issue here should now be fixed, and also, node 11 support is in.

@youpiyoful
Copy link

Same problem, i fix him just with sudo... strange i think it was not a permission problem... don t understand

@nicolasnoble
Copy link
Member

Your issue is most likely unrelated to what was going on here. Please open a new issue with the full report of the failure.

@grpc grpc locked as resolved and limited conversation to collaborators Nov 20, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests