Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prebuilt binaries #46

Open
coderaiser opened this issue Mar 5, 2017 · 28 comments

Comments

@coderaiser
Copy link
Contributor

commented Mar 5, 2017

Would be great if node-pty use something like prebuild to avoid compiling on every install. Especially on windows. Alot people will appreciate it.

@Tyriar

This comment has been minimized.

Copy link
Member

commented Mar 5, 2017

While I'm not the biggest fan of including pre-built binaries it really makes sense particularly for winpty which will never change and takes quite a while to compile. I believe that is also the lib that has the dependencies that causes issues.

I'd rather not build and ship a binary for every single version of node though.

@Tyriar Tyriar added the enhancement label Mar 5, 2017

@Tyriar

This comment has been minimized.

@Tyriar

This comment has been minimized.

Copy link
Member

commented Mar 8, 2017

Remaining work:

  • Both ia32 and x64 architectures of winpty need to be supported depending on the requested build type and linked correctly in binding.gyp.
  • Get it working in VS Code, last time I checked ./demo/fork_demo.js worked fine but when I tried to include it in VS Code it wasn't working.
  • Figure out how to best clean the node-pty module binaries so that only the current arch is included on install. Downloading the relevant binaries might be the best idea, I tried to programmatically extract the zip from https://github.com/rprichard/winpty/releases but all node unzip modules failed to parse the release zip file. This is a requirement of VS Code's build system.
@stevenvachon

This comment has been minimized.

Copy link

commented Jul 14, 2017

Why not use is-windows to decide whether to run npm install windows-build-tools as a post-install script?

@Tyriar

This comment has been minimized.

Copy link
Member

commented Jul 14, 2017

The build is already too long imo, the better fix is to remove the build all together as it's not necessary since nothing in the winpty lib is modified.

@stevenvachon

This comment has been minimized.

Copy link

commented Jul 14, 2017

Can this maybe be pushed to 0.8.0 so that 0.7.0 can be released? My Travis CI builds are failing.

@Tyriar

This comment has been minimized.

Copy link
Member

commented Jul 14, 2017

@stevenvachon why not run npm install windows-build-tools before installing on Travis?

@Tyriar

This comment has been minimized.

Copy link
Member

commented Jul 14, 2017

Wait, Travis support Windows now? Do you mean appveyor?

@stevenvachon

This comment has been minimized.

Copy link

commented Jul 14, 2017

I'm running both Travis and AppVeyor. I'm using v0.6.x of this lib, which does not have #66, so my Travis builds are failing.

@Tyriar

This comment has been minimized.

Copy link
Member

commented Jul 14, 2017

@stevenvachon for Linux/macOS builds like https://travis-ci.org/stevenvachon/dotenv-prompt/jobs/253712264 you can copy this config which runs install fine: https://github.com/Tyriar/node-pty/blob/master/.travis.yml

Your build is failing because of nan's compiler requirements:

../node_modules/nan/nan.h:43:3: error: #error This version of node/NAN/v8 requires a C++11 compiler

Your windows build should be fixed by installing windows-build-tools

@Tyriar Tyriar modified the milestones: 0.8.0, 0.7.0 Jul 15, 2017

@daviwil

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

We needed prebuilt node-pty for the atom-terminal-tab package so I took @coderaiser's advice and tried using prebuild to generate prebuilt packages. Turns out that it works really well! I created another npm module called node-pty-prebuilt which serves up these prebuilt binaries in a virtually transparent way (even with the winpty bits on Windows). Here's some more info:

https://github.com/daviwil/node-pty-prebuilt/blob/prebuild/README.md

@Tyriar (hey, by the way! 😄) if you're interested in eventually making this an official part of node-pty, I'd be happy to make a PR once we've seen it work well for a while. If not, I plan to keep the prebuilt package up to date with the latest node-pty releases so that they can both be used interchangeably.

@coderaiser

This comment has been minimized.

Copy link
Contributor Author

commented Jan 16, 2018

I'm using node-pty with gritty and node-pty-prebuilt reduces installation time drastically on linux:

Install using node-pty:

+ gritty@1.4.25
added 96 packages in 40.985s

Install using node-pty-prebuilt:

+ gritty@1.4.26
added 1 package in 6.931s

Would be great if node-pty use prebuilts this way :).

@Tyriar

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

@daviwil 👋. This is exactly the sort of thing I'm after, the Windows build in particular takes quite a while to build due to all the extra winpty code. Some questions:

  1. How do the assets get uploaded to https://github.com/daviwil/node-pty-prebuilt/releases? Do you have to build on every machine type in order to upload the assets? Seems like a lot of work 😅
  2. How do you get builds for all those different v8 versions?
  3. When a new Electron release happens would we need to upload new assets to get it working?
@daviwil

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

  1. prebuild's --upload parameter takes care of it if you provide it with a GitHub API token that has write access to the repo. You'd just need to add a call prebuild on your CI builds for all 3 platforms to get everything prebuilt and uploaded automatically. Here's how I do it for AppVeyor and TravisCI

  2. prebuild --all generates builds for all "supported" Node.js and Electron versions

  3. Yep, you'd have to upload a new prebuilt package for the new Electron version. You can easily upload new prebuilt packages for existing node-pty versions but it might require one-off CI runs (or running it manually on your machine). prebuild --upload will upload to the tag associated with the package.json version and will only replace the prebuilt artifacts that aren't already attached to the release

Caveat, the Windows prebuild support needs a new update of prebuild to be shipped after this PR gets merged (should happen in the next day or two).

@Tyriar

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

@daviwil and for people who want to use node-pty on a brand new version of node.js for example, is there a fallback if the prebuilt assets are not present?

@daviwil

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

Yep, it's built into node-pty-prebuilt's install command:

"scripts": {
    "tsc": "tsc",
    "tslint": "tslint src/**/*.ts",
    "install": "prebuild-install || node scripts/install.js",
... 
@daviwil

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

The prebuild PR got merged and 7.1.0 is released, here's what the Windows prebuild command looks like when including the winpty bits:

prebuild --all --include-regex "\.(node|exe|dll|pdb)$" -u <github_token>
prebuild --all -a ia32 --include-regex "\.(node|exe|dll|pdb)$" -u <github_token>

That builds x64 and ia32 builds of winpty and node-pty for all supported Node.js and Electron versions on Windows. Works the same way on Linux and macOS (x64 only) if you get rid of the --include-regex parameter.

@Tyriar

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

What I'm hearing is there's very little stuff we need to do upon release and if the desired version is not there is will fall back to regular building I say we try push this in.

Would it make sense to run prebuild --all in prepublish and prebuild --upload in postpublish?

Also this would need to be improved to work on Windows prebuild-install || node scripts/install.js (I think || breaks in cmd).

@daviwil

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

Here's what happens on Windows running npm install --verbose in a package that takes a dependency on node-pty-prebuilt when a prebuilt package can't be found for the platform:

...

> node-pty-prebuilt@0.7.3 install C:\dev\atom-terminal-tab\node_modules\node-pty-prebuilt
> prebuild-install || node scripts/install.js

prebuild-install info begin Prebuild-install version 2.4.1
prebuild-install info looking for local prebuild @ prebuilds\node-pty-prebuilt-v0.7.3-electron-v53-win32-x64.tar.gz
prebuild-install info looking for cached prebuild @ C:\Users\daviwil\.atom\.apm\_prebuilds\https-github.com-daviwil-node
-pty-prebuilt-releases-download-v0.7.3-node-pty-prebuilt-v0.7.3-electron-v53-win32-x64.tar.gz
prebuild-install http request GET https://github.com/daviwil/node-pty-prebuilt/releases/download/v0.7.3/node-pty-prebuil
t-v0.7.3-electron-v53-win32-x64.tar.gz
prebuild-install http 404 https://github.com/daviwil/node-pty-prebuilt/releases/download/v0.7.3/node-pty-prebuilt-v0.7.3
-electron-v53-win32-x64.tar.gz
prebuild-install WARN install No prebuilt binaries found (target=1.6.15 runtime=electron arch=x64 platform=win32)

C:\dev\atom-terminal-tab\node_modules\node-pty-prebuilt>if not defined npm_config_node_gyp (node "c:\Users\daviwil\AppDa
ta\Local\atom\app-1.23.3\resources\app\apm\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.j
s" rebuild )  else (node "c:\Users\daviwil\AppData\Local\atom\app-1.23.3\resources\app\apm\bin\\..\node_modules\node-gyp
\bin\node-gyp.js" rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  Agent.cc
  AgentCreateDesktop.cc
  ConsoleFont.cc
  ConsoleInput.cc
  ConsoleInputReencoding.cc
...

After some testing, it looks like || does work on Windows. You can test this by creating a file called test.cmd with these contents:

echo Test!
exit /b 1

Then in cmd.exe run this:

cmd /C test.cmd || notepad

When the .cmd file returns a non-zero error code, Notepad will be launched. I only have a Windows 10 machine to test on at the moment so it might be worth testing that on a Win7 machine if you have one handy.

Regarding running prebuild inside of prepublish/postpublish, that might be cool but you'd only be able to do that on one platform (since you can only publish the release once). If you want to do this for all 3 platforms you'll probably need to do it as part of a CI script.

@daviwil

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

By the way, if you want to give this a shot, I'm happy to send a PR :)

@Tyriar

This comment has been minimized.

Copy link
Member

commented Jan 18, 2018

After some testing, it looks like || does work on Windows. You can test this by creating a file called test.cmd with these contents:

Does this depend on the shell that's running or does it work on all? I'm up for adding gulp if that's better.

If you want to do this for all 3 platforms you'll probably need to do it as part of a CI script.

@daviwil oh ok,

By the way, if you want to give this a shot, I'm happy to send a PR :)

👍, if you add this I can add you as a repo collaborator if you want to own the publishing of these assets?

@daviwil

This comment has been minimized.

Copy link
Member

commented Jan 19, 2018

I don't think the shell that's running matters, I believe it shells out to cmd.exe always. powershell.exe is the default on Windows 10 and npm is still shelling out to cmd.exe as far as I can tell.

I'll definitely give this a shot this weekend! And I'm happy to maintain the prebuild/publishing aspect if you like, it's pretty hands-off after the initial work is done.

One thing I realized when thinking about this: node-pty uses a tag format that isn't (currently) supported by prebuild. prebuild expects that release tags will be prefixed with a v while node-pty omits the prefix. Since I wouldn't want you to have to change your tag naming scheme, I filed issue prebuild/prebuild#196 to ask the prebuild folks if they're interested in providing an option for this. If they're good with it, I'll send them a PR before I start migrating the prebuild work over to node-pty.

@jianyunt

This comment has been minimized.

Copy link

commented Feb 12, 2018

Hi @daviwil :) glad to see you here. I have somewhat related question. I may have to modify winpty.cc to let it take user cred. After modifying it, I need to compile it, drop it to right location etc. to run node-pty. Yes this is for the cloudshell stuff. How do you compile these cc files? I am on Win10 or WS2016. Thanks!

@Tyriar

This comment has been minimized.

Copy link
Member

commented Feb 12, 2018

@jianyunt you should try upstream this to https://github.com/rprichard/winpty if the change is of sufficient quality. Also see that project for more details on compilation.

@jianyunt

This comment has been minimized.

Copy link

commented Feb 13, 2018

Sure thanks @Tyriar . BTW, I figured it out. The easy way to build winpty stuff on windows is here.

@Tyriar

This comment has been minimized.

Copy link
Member

commented Feb 14, 2018

@jianyunt do you mean windows-build-tools on npm?

@jianyunt

This comment has been minimized.

Copy link

commented Feb 14, 2018

Yes. Here are 4 lines to build it on my local box.

npm install -g node-gyp
npm install --global --production windows-build-tools
cd c:\<myfolder>\node_modules\node-pty
node-gyp.cmd to rebuild
@Tyriar

This comment has been minimized.

Copy link
Member

commented May 4, 2018

Copying over my update on this from #179

I've been thinking about this a bit recently and don't want to proceed for the following reasons:

  • N-API should fix the majority of the issues with prebuilds, once Electron reaches node 10 we can start publishing single binaries per platform to GitHub releases or something with much more ease
  • Builds will take much longer when targeting so many platforms
  • I just moved the project over to the VSTS public projects preview which renders a bunch of this PR invalid
  • prebuild and prebuild-install are non-dev dependencies (ew)

In the meantime I expect @daviwil's https://github.com/daviwil/node-pty-prebuilt to stay up to date with the very few node-pty releases that will go out. @amejia1 sorry about the hassle.

@Tyriar Tyriar removed this from the 0.8.0 milestone May 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.