New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows 64-bit builds. #9218

Merged
merged 17 commits into from Oct 14, 2017

Conversation

Projects
None yet
4 participants
@abernix
Member

abernix commented Oct 12, 2017

This supersedes and builds on @hwillson's #9173. For full details, consult that issue for some backstory.

This PR contains numerous changes to the Windows "dev bundle" to support 64-bit builds. Previously, all Windows users (even those on 64-bit capable platforms), have used 32-bit flavors of Node.js and Mongo (plus other build dependencies, such as 7z and Python).

While the original intention was to support Mongo 3.4, this continues to use Mongo 3.2. First and foremost, it's important to ensure that Windows 64-bit builds are working before enabling Mongo 3.4, which will be slated for a future release.

This should hopefully also resolve an issue with asynchronous stack traces not appearing on 32-bit Windows during use of the Node+V8 Inspector API, presumably because it only utilizes 64-bit integers to identify async operations.

Since the existing Meteor (on Windows only) will have no knowledge of a 64-bit capable Meteor world, this will require an update to the installer and a reinstall for Windows users. More on that soon, but it should be well worth the upgrade since Meteor will no longer need to run within the 32-bit WoW64 subsystem on 64-bit Windows boxes.

Notable:

  • Enable 64-bit builds in generate-dev-bundle.sh (0cce6b5)
  • Python 2.14.12 updated to 2.14.14 + 64-bit 7z added to S3 download. (acbe9ac)
  • Removed duplicated Node headers, instead opting to point node-gyp at the headers included in <dev-bundle-root>/include/node. (f108053)
  • generate-dev-bundle.ps1 is now ~40% faster. (e4f6630)

Todo: Update History.md to mention that a reinstall will be necessary to take advantage of 64-bit, once that's figured out.

@abernix abernix requested review from benjamn and hwillson Oct 12, 2017

@benjamn benjamn added this to the Release 1.6 milestone Oct 12, 2017

@@ -0,0 +1,27 @@
<#
.Synopsis

This comment has been minimized.

@benjamn

benjamn Oct 12, 2017

Member

You truly are becoming a PowerShell master, like it or not.

This comment has been minimized.

@abernix

abernix Oct 12, 2017

Member

🔌 🐢 ⚡️

// of the PROCESSOR_ARCHITEW6432 environment variable to determine if the
// Windows architecture is 64-bit, then convert to a "uname -m" matching
// architecture label (e.g. i386, x86_64).
exports.architecture = () => {

This comment has been minimized.

@benjamn

benjamn Oct 12, 2017

Member

How about just

export function architecture() {

?

@@ -99,7 +99,7 @@ dir node
# node-gyp (the tool that rebuilds binary node modules). #WinPy
cd "$DIR"
$py_s3_url = "https://s3.amazonaws.com/com.meteor.static/windows-python/python-${PYTHON_VERSION}.7z"
$py_s3_url = "https://s3.amazonaws.com/com.meteor.static/windows-python/$PLATFORM/python-${PYTHON_VERSION}.7z"

This comment has been minimized.

@benjamn

benjamn Oct 12, 2017

Member

I noticed the python-2.7.12.7z archive is gone now. Any reason to preserve that?

This comment has been minimized.

@abernix

abernix Oct 12, 2017

Member

Should still be present? https://s3.amazonaws.com/com.meteor.static/windows-python/python-2.7.12.7z

There was no pressing reason to destroy it.

History.md Outdated
@@ -25,6 +16,12 @@
version of `npm` used by `meteor npm ...` commands.
[PR #8835](https://github.com/meteor/meteor/pull/8835)
* Windows now supports a 64-bit architecture using native 64-bit binaries.
In addition to hopefully being faster, this should allow Windows users to take
better advantage of the new V8 inspector API, which appears to only support

This comment has been minimized.

@benjamn

benjamn Oct 12, 2017

Member

Let's remove "hopefully" and change "appears to only support" to "only supports."

This comment has been minimized.

@abernix

abernix Oct 12, 2017

Member

Ok! Sticking to the firm facts!

History.md Outdated
@@ -42,6 +39,12 @@
[Feature Request #194](https://github.com/meteor/meteor-feature-requests/issues/194)
* To support developers running on a 32-bit OS, Meteor now supports both 32

This comment has been minimized.

@benjamn

benjamn Oct 12, 2017

Member

Put a hyphen after 32: "both 32- and 64-bit versions"

This comment has been minimized.

@benjamn

benjamn Oct 12, 2017

Member

Am I copy-editing English prose because PowerShell is so foreign to me? Perhaps.

This comment has been minimized.

@abernix

abernix Oct 12, 2017

Member

PowerShell approved verbs are prose, of sorts. 🤕

Mind you, I only followed them loosely.

abernix added a commit that referenced this pull request Oct 12, 2017

@abernix

This comment has been minimized.

Member

abernix commented Oct 12, 2017

Feedback all addressed! Thanks! Happy to adjust further as well.

@hwillson

This is awesome @abernix! Everything looks great - LGTM!

* a benefit to using 64 bit Node on Windows, and 32 bit works properly
* even on 64 bit systems.
* os.windows.x86_64
* Once, on the far side of yesterday, there was not a 64-bit

This comment has been minimized.

@hwillson

hwillson Oct 13, 2017

Member

More story-telling comments like this please! 🍿

hwillson and others added some commits Oct 4, 2017

Add Tool support for both 32 bit (3.2) and 64 bit (3.4) Mongo
These changes introduce dual Mongo support into the Meteor
Tool. 32-bit Mongo (3.2.15) will be used by Meteor when the
Tool is run on a 32-bit OS (32-bit Linux and Windows). 64-bit
Mongo (3.4.9) will be used when the Tool is run on a 64-bit
OS (64-bit Linux, Windows and macOS).

Fixes meteor/meteor-feature-requests#129.
Introduce `os.windows.x86_64` architecture for 64-bit Windows.
This commit reverts much of the work @hwillson had put in place for
#9173, which made it possible for 32-bit and 64-bit
Mongo versions to exist in harmony within the same dev-bundle.  That
hard work was not in vein though as it offered invaluable research.
Ultimately, this showed that a more aggressive approach would be ideal,
even if the proposed option would have worked great in the short-term.

In the wake of the news that Mongo would no longer be supporting 32-bit
versions, these changes are important so 32-bit users of Meteor can
continue to have a functioning Mongo binary in development, while still
allowing Meteor to ship newer Mongo (e.g. 3.4+) binaries for 64-bit
users.  This is particularly relevant for Windows users, who have
previously only had 32-bit Meteor builds and represented a majority of
"32-bit" development, despite the fact most of their hosts supported
64-bit.  During another time in Meteor's life, this made sense.

This commit takes improved functionality to the next level (and makes
the aforementioned changes obsolete) by introducing support for building
and shipping Meteor for Windows in a 64-bit flavor (in addition to
32-bit), which will hopefully solve a number of performance matters on
Windows by using binaries which are pre-compiled for 64-bit, rather than
forcing the Windows kernel to deal with 32-bit binaries.  While Windows
has shown it's quite capable of dealing with such a situation, it seems
more and more clear (given improvements in underlying dependencies) that
performance gains could be had by freeing Windows of its 32-bit work.

This commit also further perpetuates the "archinfo" plot-line with more
story (about Windows) and various spelling corrections.
Use Mongo 3.2.15 for 64-bit, for now.
The Windows 64-bit support is believed to be important enough to be
included in  Meteor 1.6 since it fixes Meteor 1.6 issues on Windows.

While the original reason for 64-bit may have been Mongo, we'll leave
this as the 3.2.15 (but 64-bit!) version for now.
Fail the Windows dev bundle build when an error occurs.
If any part of the dev bundle generation fails, the script should stop.

I was encountering a barely-noticed failure in the Python extraction
when the download tarball was 404-ing (due to my own fault), but the
script proceeded as if the failure hadn't happened, resulting in a
broken dev bundle.
Re-work Windows "Generate Dev Bundle" process.
This is a re-write of the generate-dev-bundle.ps1 script, which occurred
during debugging of an unrelated concern of the (new) 64-bit Windows
build on our Jenkins server.  Ultimately, I'm afraid this script doesn't
solve the problem I originally set out to fix, which was a Windows
long-file path concern.

The hunch behind that thinking was that the use of npm@1 to install
npm@5 could be causing problems, since npm@1 had no concept of nested
node_modules directories.  We had used npm@1 because Node.js
for Windows hasn't always offered (via nodejs.org/dist/) versions
including npm which we could use to install our own requirements.
Happily, that is no longer the case!

While this script now deals with long paths much more gracefully itself,
I'm not sure it completely quelled the long-path issue, and there are
still some directory trees which are longer than I think they should be.

The warnings I was seeing may not have harmed the actual bundle and were
more problematic for this build script itself when it tried to deal with
the aftermath of all those files, since native Windows commands struggle
to deal with long file paths (when cleaning up, etc.).

In the end, this script does have performance enhancements though! For
starters, it's nearly twice as fast at finishing.  Most of this was
gained by avoiding back-and-forth moving of large file structures,
opting instead to directly install into the targets when possible.

It also ensures that the npm build cache is not bundled, which started
occurring since our modification of the $HOME and $USERPROFILE variables
led npm@5 to think the npm cache was in the root of the bundle.

Additionally, it no longer modifies the $PATH, in any way, during the
build. This became particularly helpful during testing when I found that
PowerShell maintained that $PATH in the environment of the host shell.

I'd like to say it increases readability of the script, which had
become a bit of a patchwork quilt, but that's YTBD and YMMV.

This is my first "complete" PowerShell script myself so it probably
still leaves some things to desired, formatting wise.  Functionality-
wise, I hope it's improved.
Avoid ConvertFrom-Json Cmd-let, unsupported in PowerShell 4.
I thought it was nifty to use this Cmd-let from PowerShell, to directly
parse the JSON output from Node's `process.release`, but as it turns
out, that wasn't supported by the version of PowerShell on our Jenkins
server.  Alas!
Remove Add-ToExistingPath.
The Generate Dev Bundle process no longer requires any modifications
to the $PATH, preventing environmental artifacts which pile up when
running the script over and over again.

@benjamn benjamn force-pushed the abernix/enable-windows-64-builds branch from 02898ea to d2a07ed Oct 14, 2017

@benjamn benjamn merged commit 0cb2b41 into release-1.6 Oct 14, 2017

1 of 4 checks passed

ci/circleci: Get Ready CircleCI is running your tests
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
continuous-integration/travis-ci/push The Travis CI build is in progress
Details
CLA Author has signed the Meteor CLA.
Details
@lhz516

This comment has been minimized.

lhz516 commented Oct 15, 2017

How do I update or reinstall the Meteor installer? I'm still not able to use 1.6-rc.10 after uninstalling and reinstalling Meteor on Win10-64x.

@abernix abernix deleted the abernix/enable-windows-64-builds branch Nov 8, 2017

abernix added a commit that referenced this pull request Nov 20, 2017

Use Mongo 3.4 for 64-bit platforms.
Most of the work to prepare for this change was done through the
excellent work of @hwillson in #9173 which, after some
re-working to support the 64-bit architecture on Windows platforms,
landed in #9218, making this change as simple as bumping
the minor version number (and rebuilding the dev bundle).

From this point forward, and due to Mongo's discontinuation of 32-bit
support in newer versions of MongoDB, 64-bit platforms will, in
development, use newer versions of Mongo while 32-bit architectures
will remain at 3.2.x versions.

Of course, in production, apps are free to use whichever version of
Mongo they would like, provided that version is supported by the
Node Mongo driver and Meteor's Mongo data packages.  At this time
there is no target for when Meteor will stop supporting Mongo 3.2,
but developers are encouraged to take steps to upgrade their Mongo
deployments (through their database providers) to newer versions
since Mongo has set September 2018 as the "End-of-Life" for Mongo
3.2.x.  For more information on Mongo support cycles, see their
support documents at https://www.mongodb.com/support-policy.

Refs: #9173
Refs: #9218

abernix added a commit that referenced this pull request Nov 21, 2017

Use Mongo 3.4 for 64-bit platforms.
Most of the work to prepare for this change was done through the
excellent work of @hwillson in #9173 which, after some
re-working to support the 64-bit architecture on Windows platforms,
landed in #9218, making this change as simple as bumping
the minor version number (and rebuilding the dev bundle).

From this point forward, and due to Mongo's discontinuation of 32-bit
support in newer versions of MongoDB, 64-bit platforms will, in
development, use newer versions of Mongo while 32-bit architectures
will remain at 3.2.x versions.

Of course, in production, apps are free to use whichever version of
Mongo they would like, provided that version is supported by the
Node Mongo driver and Meteor's Mongo data packages.  At this time
there is no target for when Meteor will stop supporting Mongo 3.2,
but developers are encouraged to take steps to upgrade their Mongo
deployments (through their database providers) to newer versions
since Mongo has set September 2018 as the "End-of-Life" for Mongo
3.2.x.  For more information on Mongo support cycles, see their
support documents at https://www.mongodb.com/support-policy.

Refs: #9173
Refs: #9218
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment