Hi, I'm using node-serialport in MicroFlo and to avoid the end-users needing a C++ compiler I ship a pre-build archive that they can use, including compiled version of node-serialport (the only C++ dependency).
I am currently using a some ugly hack for this:
It looks to me that using https://github.com/springmeyer/node-pre-gyp, one could solve this problem in a much nicer way.
This looks great! Glad to see someone step up to solve this challenge for node (especially for Windows users).
@jonnor are you offering to sponsor the s3 hosting or is your request that someone from the node-serialport team sponsor the s3 hosting costs?
@JayBeavers I have not given hosting any thought when I filed the request. The node-pre-gyp docs state that any hosting can be used though. So maybe just push the built artifacts from Travis to a separate git repo here on github?
To play nice with github, probably only store tagged/released versions.
I'm happy to host the s3 bucket.
This looks like we might want to start using git tags and having travis start publishing to both npm and s3 for the binaries. It would not be trivial to setup but would be really cool. I'm happy to help but @jonnor you should lay the groundwork. =)
to using tags. to node-pre-gyp. What's happening with this?
For MicroFlo I just moved over (this weekend) to distribute a Chrome app to end-users for serial communication, primarily because it can also provide a UI easily.
Still using node-serialport for embedded systems but there pre-built is not important. So from my perspective, this is no longer an issue.
We're working on this problem actively and hope to get something in place using node-pre-gyp perhaps even as soon as end of this week, so we can submit a coherent PR. Getting setup with auto-builds from Travis & AppVeyor are the final step to fully automating the problem away.
@reconbot we'd love your insight/help into this process as my colleague @edgarsilva works on it, if you have time this week.
@deadprogram I'm around to help as well.
We already have linux x86 32bit, 64bit and OSx binaries being build, packaged and published. There is an issue though with nodejs v11 in the 32 bit version.
When travis runs npm test one of the tests fails, you can check the details here:
Packages are built correctly and published, but the travis job fails because of it. What do you think guys? any suggestions on how to handle it?
Also, we're going to work on the automation of the Windows builds tomorrow.
@everyone I'm trying to setup appveyor.yml to automatically generate the windows binaries, same setup as travis.yml, but so far appveyor.com docs are pretty lacking. Does anyone here has any experience with that? or any suggestions?
@BergWerkGIS is familiar with appveyor and can likely assist. How far did you get @edgarsilva?
I sent a couple links to @edgarsilva already, but any help from someone with more experience is appreciated. @BergWerkGIS please feel free to chime in!
@deadprogram, I'll be happy to take a look with you today as well.
@springmeyer just setup and jobs start but fail, I have not put any node-serialport specifics yet in the yml file since the job was failing just printing env variables.
I'm getting more progress based on the links @deadprogram sent me though.
First successfull build is here:
Working on appveyor.yml.
Just a few minutes!
@BergWekGUS nice, what appveyor.yml file are you using?
You can find the full appveyor.yml docs here:
Here's the one, I've hacked together quickly:
For different node versions on Windows, I'm using nodist:
I'll be offline during the weekend, but I can assist with node-pre-gyp next week.
@BergWerkGIS awesome, I see it running I can continue working with that and let you know if anything comes up or have any questions.
@edgarsilva you are welcome.
OK thanks to all this good collective effort, we're just about ready to submit 3 PRs:
Thanks to everyone for helping make this happen, soon no more crazy install problems at hack sessions!
@voodootikigod can you please create a osx-node-pre-gyp branch off of the master branch, so we can submit the second of the three PRs?
@everyone We are packaging and publishing the windows x86 32bit binary, but for some reason nodist always detects the platform architecture as being 32bit, We have a workaround using what @BergWerkGIS suggested with nodist and forcing the 64 node version in appveyor, but this causes a problem on compilation:
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.Cpp.InvalidPlatform.Targets(23,7): error MSB8007: The Platform for project 'serialport.vcxproj' is invalid. Platform='x64'. You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Platform that doesn't exist for this project. [C:\projects\node-serialport\build\serialport.vcxproj]
What do you think guys? any suggestions/comments on how to solve the platform arch issue in appveyor?
I'm not sure about the nodist/appveyor specifics, but one potential workaround is to use the --target_arch flag to node-pre-gyp. This allows you to build and package a 32 bit binary (--target_arch=ia32) from a 64 bit system.
@springmeyer that is kind of what we need, but the other way around, we are working off a x64 platform, the ia32 version gets compiled and packaged ok, but the x64 fails, this is the error:
I'm going to give it a try with the --target_arch flag.
@springmeyer that is exactly what I think is happening since I was having a similar problem in one of my windows machines till I updated the SDK and Visual Studio express 2013.
@everyone FYI: I'm working with the owner of AppVeyor on this issue.
Will report here, when there is progress.
#319 is amazing. =)
And of course a large change to how it gets built and published.
Linux binaries will get built on travis (every time?)
Windows binaries will get built on appveyor (every time?)
What's the reason for the separate deploy branch for osx? And when does that get deployed? (I'd rather apply a patch or use a separate set of files, the less the CI commits back to the repo the better)
Lastly I've been saying "deployed" but is that just a binary for each sha tracking on master? Are we publishing to npm each time?
@edgarsilva can explain better, but the separate OS X branch is required since that branch has to be built as Objective-C with Clang, and (to my knowledge) there's no way to run Travis for multiple languages in the same build.
@stewart has an accurate description. The current build pattern used by many modules that do multi-platform builds usnig node-pre-gyp that include OSX, is to include a separate build branch.
For examples, take look at https://github.com/mapbox/node-sqlite3 and https://github.com/mapbox/node-blend
As far actual release process, the make task does the branch massage, and equally importantly uses tags to make sure that the source repo & the released code match up. Since the binaries are named based on the version, then the binaries downloaded during install match up to the node-serialport version being installed.
Hopefully that makes sense.
@edgarsilva @springmeyer x64 Windows build works! YEAH!!!
Took some time for me to figure out, but the fix is very easy.
The problem was, that node-gyp is looking for MSBuild.exe in the path and then searches registry.
Unfortunately .Net Framework MSBuild.exe 4.x is found first in the path. So, no check in registry.
That's why native C++ MSBuild.exe 12.x is not found.
.Net Framework MSBuild.exe 4.x
native C++ MSBuild.exe 12.x
So the fix is just adding the MSBuild12 directory to the path and telling node-gyp to generate a VS2013 solution file:
Proof of successfull x64 build:
@BergWerkGIS great! that covers all the 3 big platforms in both architectures!!! I'll add those lines to the appveyor.yml file.
Basically what I'll do is this:
@BergWerkGIS what do you think?
@reconbot what @deadprogram describes is accurate, let me elaborate a bit more:
Linux binaries will get built on travis (every time?) (Only when tagging for a release or if the commit message contains [publish binary] )
Windows binaries will get built on appveyor (every time?)( Same as previous question.)
What's the reason for the separate deploy branch for osx? And when does that get deployed? @stewart description is right on the mark, the make release task created by @stewart does the branch release massaging as explained by @deadprogram.
Lastly I've been saying "deployed" but is that just a binary for each sha tracking on master? Are we publishing to npm each time? the binaries are for each node-serialport version, for each of the node.js versions supported e.g.
node-v11-darwin-ia32.tar.gz Standard 13.8 KB Thu Apr 03 20:31:34 GMT-600 2014
node-v11-darwin-x64.tar.gz Standard 13.6 KB Thu Apr 03 20:31:20 GMT-600 2014
node-v11-linux-ia32.tar.gz Standard 15.7 KB Thu Apr 03 19:21:50 GMT-600 2014
node-v11-linux-x64.tar.gz Standard 17.1 KB Thu Apr 03 19:21:27 GMT-600 2014
node-v11-win32-ia32.tar.gz Standard 63.2 KB Fri Apr 04 20:35:22 GMT-600 2014
node-v14-linux-ia32.tar.gz Standard 17 KB Thu Apr 03 19:41:18 GMT-600 2014
node-v14-linux-x64.tar.gz Standard 18 KB Thu Apr 03 19:40:57 GMT-600 2014
v8-3.11-darwin-ia32.tar.gz Standard 13.7 KB Thu Apr 03 20:30:19 GMT-600 2014
v8-3.11-darwin-x64.tar.gz Standard 13.5 KB Thu Apr 03 20:30:05 GMT-600 2014
v8-3.11-linux-ia32.tar.gz Standard 15.8 KB Thu Apr 03 19:22:22 GMT-600 2014
v8-3.11-linux-x64.tar.gz Standard 17 KB Thu Apr 03 19:21:58 GMT-600 2014
Let us know what you think.
@edgarsilva looks good.
This is good stuff. What's the "release tag" look like?
@stewart can expand on this further, but it is just the node-serialport version in the package, as follows:
@git tag -m "$(VERSION)" v$(VERSION)
All of this handled in the make release task.
amazing work everyone!
Thanks to @BergWerkGIS now we also have pre-built binaries for win x64! all major platforms, both architectures for all of them! thanks everyone!
We are now just waiting for the PR to be merged and the osx-node-pre-gyp branch created so we can merge our osx build branch.
Seems like we're ready.
Steps to merging this and release should be:
git fetch origin
git checkout master
git checkout -b osx-node-pre-gyp
git push origin osx-node-pre-gyp
git checkout master
Now all will be merged and ready.
How to release new versions of node-serialport:
Edit the package.json to change to the new version number. Commit to master. Now run the release task:
@voodootikigod nice, let me merge the osx branch into that one and delete all the binaries so the new ones are generated automagically.
Just deleted all of the binaries from the bucket so we can check they are generated on release.
@voodootikigod not sure if node-serialport already has appveyor CI setup.
tiwtter --> @papichinox ...
Same as travis, signup:
and enable CI for that repo in github.
TravisCI and Appveyor failed.
Can you paste the job links pls?
@voodootikigod it should handled by the appveyor.yml, you just need to setup the repo to be picked up, but it seems that is done already... Same thing happened to me when the secure keys were incorrect, I wonder if they are bound to an account...
@voodootikigod looks like same thing with travis, secure keys are not correct for some reason. Reviewing them now.
@voodootikigod after checking the repo it seems it is a security issue with the bucket access. Checking on hot to fix now.
Found couple of things.
I think appveyor job fails because encrypted secure keys might be account bound, @BergWerkGIS do you have any idea if this is true? the job is behaving exactly as when I had one of the secure env keys with a typo.
The osx make release branch needs a small tweak, we forgot to include the [publish binary] message so it publishes the binaries to the bucket.
Lastly and not sure why, it seems travis is not picking up the secure environment variables, we just updated our fork with the current main node-serialport repo and I'm checking this issue.
@stewart is about to publish the first small tweak to the make release task and I will update for the other 2 ASAP.
@voodootikigod appveyor issue was just confirmed by support team, secure env var are account bound, so we'll have to provide you with keys for you to encrypt in the appveyor sire and update the appveyor.yml file.
Still checking the travis environment variables not being set issue.
@voodootikigod just confirmed that travis secure keys work in the same way, they are repo/account bound so they will not work in PRs, let me know when you have some time, I can provide you with the keys and help you with encryption process if you are not familiar with it.
@BergWerkGIS you might be able to help with this small issue, I'm seeing this warning in couple of the commands in appveyor log but I was not able to get rid of them:
Do you have any idea what that's about?
@edgarsilva Interesting: This is a Powershell error and as far as I know neither node-gyp nor node-pre-gyp use Powershell. Maybe it stems from how AppVeyor executes the cmds.
Only thing I can think of at the moment, is trying to remove 2>&1 from these lines IF %PUBLISH_BINARY%==true (node-pre-gyp package publish 2>&1).
We had to use these before, because npm, node-gyp, ... write to StdErr instead of StdOut and that was treated as error by AppVeyor and stopped the build. But I think in the meantime this has been dealt with.
IF %PUBLISH_BINARY%==true (node-pre-gyp package publish 2>&1)
npm, node-gyp, ...
If removing 2>&1 doesn't help, I would suggest contacting AppVeyor support.
@BergWerkGIS thanks for the headsup, I will give it a try after finishing with the secure keys issue, and see what I get with your suggestion, thanks.
@everyone good news, we are now building packaging and publishing successfully!!! for all 3 major platforms windows, OSx and Linux, both architectures ia32 and x64/
Thanks @everyone! Great Work!
We still need to do a couple of small tweaks to the travis file, but that is for travis itself not the publishing of binaries on update.
This has been the best ticket since windows support! I am happily closing this unless someone reports a reason not to.
The final step, is for a committer who has rights to npm to update the package.json version, push it, and run make release. Make sure to update the package, so the release tagging works.
@voodootikigod checking them, this in particular seems to be an issue with serial port and 32bit nodejs:
What do you think?
Working on blacklisting publishing binaries for node 0.9.x; since version is reported the same for node 0.10.x
@voodootikigod the issue with the duplicated binary has been fixed, there is still one test failing with node.js v0.11, not sure if that was a known issue, but couldn't pin point the root cause. It does not seem related to the binaries:
Maybe it would be easier you guys to give it a glance since you are more familiar with the code.
Can you rerun? I think it's a mocha issue.
I take that back, rerun fails too.