A *plea* to include binary API required by socket.io into NodeJS core.

[tohagan]

The Problem

There is unending pain on Windows for anyone attempting to use the beloved socket.io library. This is rooted in the fact that you're forced to perform a compile and worse in a SDK (Visual Studio 2010) most Windows devs ditched some years ago. You can kinda/maybe/possibly hack around it if you have admin privs and a lots of Window experience but basically any app that dares to use this wonderful lib is in for a ton of support issues and pain and most end users won't be able to get it working.

Just google search for "win7 nodejs socket.io" and feel the pain out there.

Today I'm hitting this issue yet again as I'm trying to use a new PhoneGap component that would be very cool with remote logging if not for this ten pound gorilla standing in my path.

It's not just Windows that experiences this pain. Other platforms also suffer but for me today it's another Windows issue and this is a recurring nightmare for anyone doing node on Windows.

The Solution

IMHO I think the right solution is to remove the need for compiling for what I think is now a frequently used CORE feature of node by adding the API required by socket.io into the node core code. So the fix then just just a Node upgrade rather than a two day torture test.

The Plea

Socket.io is FAB! It opens tons of cool trick doors for node. Please remove the pain. All the node community will rejoice.

[tohagan]

FYI ... I'm using node 0.10.29

[TooTallNate]

socket.io's only native dependency is ws, which actually has an optional compilation step that falls back to a vanilla JavaScript implementation if compilation fails.

That said, the compiled portion of socket.io is optional and only benefits speed in the end. Where are you getting hung up exactly?

[bnoordhuis]

I think what @tohagan is complaining about is that gyp / node-gyp defaults to VS 2010 when auto-detection fails and GYP_MSVS_VERSION isn't set in the environment. (Upstream gyp defaults to 2012 now, by the way.)

@tohagan You can set e.g. GYP_MSVS_VERSION=2012 or GYP_MSVS_VERSION=2013e (the 'e' stands for 'express edition') to pick a particular version of VS. See https://github.com/joyent/node/blob/v0.10.29/tools/gyp/pylib/gyp/MSVSVersion.py#L209-294 for the full list.

[TooTallNate]

I'll look into updating gyp in node-gyp ASAP.

[tohagan]

Yes .. As a developer with many years experience I can probably dodge this bullet and follow instructions on stackoverflow or here but the fact remains is that large numbers of end users of node apps and libs cannot. Yes I'm aware that ws is the problem. So I'm pleading that you include it into CORE so we can avoid the compiler issue and keep sockets fast.

There are tons of great node libs and apps that are simply very painful to deploy on Windows because of this issue and this pain has now subsisted for many months. Hacking the environment does not fix it for non tech users. Please realise that Node is now maturing and that it need to be "turn key" installed for all users - not just developers. That's the very cool dream of npm.

ws is a wonderful and essential part of NodeJS but because of the serious and constant pain this module creates for Windows deployment, I'm lobbying that it become part of NodeJS CORE library so we can finally bury the need to be compiling anything.

So .. I'm pleading ... Can you please team up with the ws team so we can all rest easy at night knowing we won't have to face another request to install VS 2010 or any other developer tool in the morning.

The fact is that now many NodeJS end users who install our node apps ...
(a) Don't have admin privs on Windows
(b) Don't have the technical expertise to dodge the ws bullet.
(c) Are Not developers
(d) Are Not Visual Studio developers
(e) Ditched Visual Studio 2010 long ago

However, these end users typically can download and install the next release of NodeJS. So I think ws and NodeJS CORE belong together - precompiled for each platform so we can forget this nightmare.

Just do a Google search for "win7 nodejs socket.io" and you'll get the idea.

[mwbrooks]

+1 Win7 support for socket.io would be incredibly helpful. Thanks for opening this discussion @tohagan!

[bnoordhuis]

You know, incorporating ws kind of makes sense in a twisted way. Websockets are a common enough use case that it makes a reasonable core addition.

[TooTallNate]

None of this addresses the fact that ws falls back to JavaScript implementations of the modules, and thus native compilation is not a requirement.

So I ask again, what is really the issue? Where are you being prevented from using socket.io?

If you do npm install socket.io, and the native compilation part fails, then socket.io should still be installed and doing require('socket.io') in a script at that point should still work. Is this not the case?

[tohagan]

Thanks @TooTallNate ... I will retest this. If it does work then perhaps the failure message on install need to switch to being a warning somehow (not simple I suspect).

Another longer term solution for Node might be to ship the GNU gcc compiler as part of Windows NodeJS so we get a more uniform compiling environment that no one needs to install separately. Perhaps it becomes part of gyp/ node-gyp. I realise that for Windows coders, VS is much better supported.

Might even be able to auto download / install mingw's port of gcc as part of the install process. Not sure if their licence will allow this. Might need to chat with them first.
http://www.mingw.org/

IMHO I still think ws belongs in CORE even with this.

[Qard]

+1 on core support for websockets.

On a side note, including ws in core doesn't solve the native compilation issue for any of the other many native modules out there. The proposed solution does not solve the actual problem, which is that compiling native modules on Windows just sucks. With Google switching to gn, the future of gyp is a bit unknown--perhaps the state of native modules on Windows will improve as we work through the deprecation of gyp.

[tohagan]

I managed to get past the issue for myself by setting GYP_MSVS_VERSION to 2012. Thanks!

[rlidwka]

Why do you even have to use it on native Windows? Just develop it using vagrant or something like that.

[dougwilson]

Another longer term solution for Node might be to ship the GNU gcc compiler as part of Windows NodeJS so we get a more uniform compiling environment that no one needs to install separately

FWIW, this is what the Perl community does for Perl on Windows (Strawberry Perl distribution).

[tohagan]

@dougwilson Very interesting. At present of course, the NodeJS community has invested some effort in Visual Studio C++ code & VS.NET projects so I guess if they went down this path you don't just drop the support for this existing code but rather, offer a migration path ... so there may be a period of time where you support both OLD (VS.NET code) and NEW (GNU gcc) and encourage projects to migrate over.

The end game is a turnkey install that any non-tech user can safely and reliably perform on any platform after having only needed to install NodeJS.

@rlidwka A Vagrant VM is serious overkill for running what is often a small JS script that would then be disconnected from the local Windows filesystem and the rest of the Window OS.

[tohagan]

Was thinking the other day that perhaps a cloud based compile of binary node packages might be the way to go as it would ensure a uniform CI based build environment and remove the need for end users to install compilers or the joyent team to include them in NodeJS CORE. The Node framework could provide this as part of the npmjs.org service (it compiles package C++ code, runs unit tests and signs binaries that can then be published to npm registry and npm client checks digital signatures of binaries.

A risk factor is that you can't confirm that source matches binary so I'm recommending this type of service uses digital signatures that include a reference to the GIT repo + version that code was built from so we can at least correlate the two. Another significant risk factor is running the untrusted unit tests in the build service.

I also found this app that looks like it's implementing some of this ... but I wonder if it really belongs as part of the npm registry?

• https://www.npmjs.org/package/node-pre-gyp

If this app is solid and security-wise safe enough, it might be viable to solve the ws problem in the short term.

[tohagan]

I've just posted this request to support Windows MinGW GCC compiler to the Google GYP team in the hope that this might help find a way forward for this issue.

• https://code.google.com/p/gyp/issues/detail?id=448

@dougwilson Very interesting to know that Perl already solved this issue using MinGW GCC. This also suggests that much of their open source C++ code may be of great value as pre-tested cross-platform code that the Node project can utilise.

[bnoordhuis]

Very interesting to know that Perl already solved this issue using MinGW GCC. This also suggests that much of their open source C++ code may be of great value as pre-tested cross-platform code that the Node project can utilise.

I think you're going to be disappointed. msvc and mingw are incompatible with one another on the C++ ABI level. That doesn't affect perl because most (probably all) modules are written in C, but it will affect node.js, where almost everything is C++.

Mandating a flag day where everyone switches to mingw is, in my opinion, not an option. For one, you'll leave everyone who uses closed source libraries compiled with msvc in the cold.

clang/llvm is steadily improving its msvc compatibility, however. It's not quit there yet, last I checked, but maybe in a year or two it can be the base of a "Node.js Windows SDK."

On the subject of sharing code between perl and node.js: most perl modules are written in perlxs, which is something of an idiosyncratic C dialect and closely tied to the perl interpreter. Chances of it being compatible with another perl implementation (if there was one) are close to zero, never mind a completely different language and run-time.

[tohagan]

I've never proposed that we instantly switch over or that we have a flag day of any kind or that we drop support for existing Visual Studio code. That's clearly unworkable. Rather I expected that it be an encouraged process so that our C++ modules can be gradually migrated to MinGW GCC so we can deploy node without Visual Studio as a dependency which is not viable for many end users and systems admins on Windows. I'm simply trying to remove a limitation for the long term so that the Node community can flourish by covering the widest audience of end users.

Yes I'm very aware that the APIs will differ. I've developed C++ code from 1987 - 2012 - mostly in Visual Studio. I wrote my first C code in 1984 and developed systems in another 59 languages in the last 33 years. Since most cross portable code needs to be developed for Unix/OSX in GCC anyway I would expect that for many projects this will be less of code maintenance burden in the longer term.

The Perl suggestion was just a passing idea with no research. Happy to be proved wrong.

[jbergstroem]

@tohagan Not super relevant to your end goal, but gyp is pretty much done with. Google is replacing it with gn and it has no plans to be decoupled from the chromium sources. All I'm saying is that you shouldn't expect to see any further development of gyp from its original authors.

[bnoordhuis]

Rather I expected that it be an encouraged process so that our C++ modules can be gradually migrated to MinGW GCC so we can deploy node without Visual Studio as a dependency

How do you see a gradual migration happening? A node.exe that is compiled with msvc won't work with add-ons that have been compiled with mingw and vice versa. Shipping two different binaries won't help with that; best case, it makes the transition less awful - but it will still be painful for some.

I mentioned clang above. I think that's the most viable solution (it's what I plan on using for cross-compiling my company's modules) and it might be usable starting with llvm 3.6, maybe even 3.5 for some source code. You can find a compatibility overview here.

[Mithgol]

@tohagan

Was thinking the other day that perhaps a cloud based compile of binary node packages might be the way to go as it would ensure a uniform CI based build environment and remove the need for end users to install compilers or the joyent team to include them in NodeJS CORE. The Node framework could provide this as part of the npmjs.org service (it compiles package C++ code, runs unit tests and signs binaries that can then be published to npm registry and npm client checks digital signatures of binaries.

A risk factor is that you can't confirm that source matches binary so I'm recommending this type of service uses digital signatures that include a reference to the GIT repo + version that code was built from so we can at least correlate the two. Another significant risk factor is running the untrusted unit tests in the build service.

I also found this app that looks like it's implementing some of this... but I wonder if it really belongs as part of the npm registry?

• https://www.npmjs.org/package/node-pre-gyp

If this app is solid and security-wise safe enough, it might be viable to solve the ws problem in the short term.

There are (already) some pretty reliable cloud-based platforms for building of Node.js binary addons, namely Travis CI for building on Linux and Mac OS X and AppVeyor for building on Windows. Both these platforms are capable of running node-pre-gyp afterwards and thus publishing the binaries.

It's OK to have doubts about whether binaries belong to npm packages. And that's why node-pre-gyp by default publishes binaries to Amazon S3 instead of npm. (Thus the npm's package contains only the module itself, while the module's binary addon is being uploaded and downloaded separately — and kept in a different storage.) The HTTPS transport to S3 (and from S3) is encrypted (as any HTTPS is) and thus at least as secure as certificate authorities are.

Downloading binaries (for end users) from other storages (i.e. not S3) is also supported by node-pre-gyp (these storages must be HTTPS hosts, for security), but publishing is not implemented, i.e. the package's author would have to use some other tool (not node-pre-gyp) to publish his (or her) binaries after they're built successfully.

(Personally I do not know of any such tool, but the idea of publishing binaries to GitHub releases looks like one of the promising alternatives because GitHub does not impose any limitations on binary files in releases, at least currently. And it's free of charge, unlike S3.)

The bottom line is: if ws uses node-pre-gyp as a (bundled) dependency, it would eliminate (for the end users) the need of building the binaries for themselves.

(As ws already uses Travis CI to build and test binaries, it would have only to start using node-pre-gyp to publish binaries from Travis after they are built and pass the tests, and then also start using AppVeyor in the same manner as Travis.)

[tohagan]

My goal in starting this conversation was not to push a particular solution but rather to find one that works for ws and all other node C++ modules. If MinGCC on Windows is looking like a brick wall then lets ditch that idea! I'm easy going and just happy that the conversation is happening :) .

Travis & AppVeyor CI cloud builds with binary installs sounds smarter and much more stable that C++ recompiling during deployment. Again .. I'd see it as a gradual transition rather that an instant change though there might be a target date or version in the Node roadmap to encourage C++ module projects to transition.

Warning ... Slight Topic diversion ...

The End Game is as I see it is that Node becomes "enterprise ready" and "home consumer ready" in the sense that node components can safely be installed by systems admins and trusted on all commonly used enterprise platforms and equally easily installed by non-techy home users. I think it's getting there.

I'd like to be able to develop both commercial and open source Node apps in the confidence that systems admins will happily embrace the end product. At the moment I can't yet have this confidence.

It's OK to have doubts about whether binaries belong to npm packages

Not sure I can agree with this one ... IMHO, SSL from S3 is a great first step but not sufficient as it assumes a specific binary deployment route (from S3) when in fact within an enterprise there can and will be many different routes. Most other languages have opted for digital signatures and with good reason as it independent of the route that the binary took prior to becoming deployed and installed. SSL only provides transport security and endpoint verification. It does not protect binary files at rest and so unless signed and verified the binary is vulnerable to modification. Legally SSL alone would not stand the test of non-repudiation or legal recognition that a digital signature offers.

It's is also why I've advocated that binaries be securely correlated to source code version using digital signatures. It just protects everyone. GIT gives us a great "leg up" here with digital signatures on commits. npm and node-pre-gyp can leverage this by offering secure correlation of source code signature with binaries.

While there's a strong case for signing binaries, I think there's also a case for signing NodeJS source code as well in correlating GIT commit =>. npm package version; though there may be other issues to consider such as performance.

End of rant

[obastemur]

The native-modules, compatibility, cross-platform etc. problem is bothering me for a long while. Unfortunately Windows still doesn't play nice with llvm-clang couple (even the latest stable 3.4.2) but It's getting there. The biggest problem isn't cross compiling though since you can compile the whole solution to llvm script and than link it to binary on the target platform. (for windows you still need the Platform SDK!! :/ )

I would have trouble choosing between vcc / clang for an application that doesn't load other compiled libraries since both have cons / pros. But vcc would be mine only choice if an app requires loading other compiled binaries. It's all about MS standard of memory registers / optimizations those complain a lot which result in unexpected crashes...

[Mithgol]

Singing binaries is probably a good idea; however, it just does not make much sense to sign only an addon when the rest of npm packages remain unsigned.

It also seems like an idea for another issue, separate from “including binary API required by socket.io into NodeJS core”.

[tohagan]

FYI ... Here's how MS do a binary install for their Node SQL Server Driver ...

https://github.com/Azure/node-sqlserver/blob/master/scripts/install.js

Sadly they failed to use HTTPS !

[jbergstroem]

Can we perhaps move this discussion to the mailing list and keep the issue tracker to issues?

[rlidwka]

Can we perhaps move this discussion to the mailing list and keep the issue tracker to issues?

Mailing list is pre-moderated, so making actual discussions takes ages there. So I'd really prefer discussing it here along with all other technical stuff (simple questions excluded - that's what stackoverflow is for).

[tohagan]

A workaround for now is to spin up a Node Docker container using Boot2Docker for Windows Docker for Windows.

Safest to select an official Node base image as the starting point for your Dockerfile. While developing, mount your Node source directory as a Docker volume so you avoid having to constantly rebuild your Docker image. When you're ready to deploy, you can use a different Dockerfile to that copies your Node source directory into a final build image (or just npm install it if it's pushed to a registry server).

Take care to ensure that your final deployment is running node as non-root user

More info:

• Docker for Windows
• https://registry.hub.docker.com/_/node/

[Sogl]

+1

[bastervrugt]

+1

[orangemocha]

This would be solved by nodejs/build#151

[Chowarmaan]

+1 - Ability to build 'out of the box' would be great.

[dsschnau]

+1

[dhasilva]

+1 for removing VS.
I don't use VS, I never used VS and I will never use VS. The only reason I have to keep this behemoth using space in my notebook is to install node modules and it does not even work properly, since I lost two work days trying to install memwatch and karma unsuccesfully.

[howardyang2009]

+1 for removing VS dependency

[fcagalj]

+1 against VS. If I wanted VS then I wouldn't use NodeJS!

[jloramas]

+1 for removing VS dependency

[LeeU1911]

I found this topic by digging through a whole lot of pages in order to solve my problem with installing mongodb via npm. It also requires VCBuild.exe in the build path, aka Visual Studio dependency.
As a Java developer, my organization obviously won't understand why I need to get a ~4GB+ copy of Visual Studio if I don't have a valid reason. This problem got worse since now I'm working on a new computer in a remote location and I have to go through all the pains with this again.

So I also vote +1 for removing VS dependency.

Sorry for the long rant.

[justinshea]

+1

[cheoAlejo]

+1

[AngelEyesChina]

+1

[akreienbring]

+1

[Marcoevich]

+1 for removing VS dependancy

[bexoss]

+1 It should be.

[jacolaisen]

+1 you shall remove VS dependancy and leave the door open!

[er-sandeepsingh]

+1

[cherryliu]

+1. Thanks @tohagan! I thought I was blocked on build error on windows too, although sockets all works well. +1 to the point that the build errors could be changed to warnings instead! Socket.io is really an amazing tool that opens so many doors, it just needs this touchup on installation UX.

[tohagan]

Was delighted this week to discover that the latest npm performs binary installs and creates a package-lock.json containing SHA checksums. Excellent work!

• https://docs.npmjs.com/files/package-lock.json
• https://w3c.github.io/webappsec-subresource-integrity/
• spec: Describe npm-shrinkwrap.json and package-lock.json npm/npm#16441

After reviewing the breaking changes in npm v5.0, I recommend you npm -g install npm today!

[Mithgol]

Does the latest npm really perform binary installs? What part of http://blog.npmjs.org/post/161081169345/v500 does mention it?

(It seems to me that the only place where http://blog.npmjs.org/post/161081169345/v500 mentions any binary is the phrase “One binary to rule them all: ./cli.js has been removed in favor of ./bin/npm-cli.js”.

That's why I ask.)

[gibfahn]

(It seems to me that the only place where http://blog.npmjs.org/post/161081169345/v500 mentions any binary is the phrase “One binary to rule them all: ./cli.js has been removed in favor of ./bin/npm-cli.js”.

That refers to the npm "binary" that you run, there's less "script calling script calling script" in npm5. It's not actually a binary (compiled) file though.

As for native modules, npm will compile them using the system C++ compiler. However a lot of modules include prebuilt binaries for most platforms, using something like node-pre-gyp. Other than having a package-lock.json that notes down the dependency versions, I don't think npm5 is doing anything different.

So if you want some package that's written in C++ to magically install without compiling, then you need the module developers to include prebuilt binaries in their npm package.

There have been a number of improvements for people using Node on Windows (many led by Microsoft).

• You can now just download the C++ compiler parts of Visual Studio (Visual C++ Build tools), there's even a one-liner that gives you everything you need to build native modules, see here.
• You can use NuGet (i.e. Chocolatey) to get the compiler
• As mentioned above, you can run in Docker containers, which gives you a Linux environment
• You can use the Windows Subsystem for Linux, which gives you Ubuntu on your Windows box

Also this repo is archived, issues with Node should be raised in https://github.com/nodejs/node. The only part of this that might be worth raising in nodejs/node is the discussion about either bundling a compiler, or moving to llvm ( @bnoordhuis how did that go?)

For general questions about node, please raise issues in https://github.com/nodejs/help instead.

I'll close this.