Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Tool support for both 32 bit (3.2) and 64 bit (3.4) Mongo #9173

Closed

Conversation

@hwillson
Copy link
Member

@hwillson hwillson commented Oct 4, 2017

(Note: I know the discussion I'm about to dive into here is likely better suited in an issue, but it relates to several issues, and ultimately leads to the PR code that is included here.)

Backstory

As discussed in meteor/meteor-feature-requests#129, the Meteor Tool is currently using Mongo 3.2.x, and it would be great to be able to upgrade to 3.4.x. As further discussed in PR #8864 however, we can't just update to 3.4.x as 32-bit versions of Mongo have been discontinued (as of Mongo 3.4). Since Meteor currently supports 32-bit Linux and Windows, we can't easily just upgrade to use a 64-bit version of Mongo.

We've talked about the best way to move forward with this, and ultimately decided that Meteor should ideally drop 32-bit support for all OS', as discussed in meteor/meteor-feature-requests#152. This would definitely make the maintenance of Meteor easier moving forward, and is in-line with the approach several other open source projects are taking (like Mongo, Ubuntu, etc.).

While dropping Meteor's 32-bit support is the ultimate goal, we have heard from some community members about how this will impact them (especially those using embedded OS'). There is also a good chunk of work to complete to make this happen, as Meteor doesn't currently offer a 64-bit version for Windows (for example). While this work is pending, developers are getting more and more frustrated by not being able to (easily) use Mongo 3.4.x features, when they're already using a 64-bit OS.

Proposal

Given the above, I think it makes sense to consider moving towards a 64-bit only Meteor in incremental steps:

  • Step 1: Update the Meteor Tool to use Mongo 3.2.x for 32-bit based operating systems, and 3.4.x for 64-bit based operating systems.

  • Step 2: Start supporting 64-bit Windows as an official Meteor architecture, and provide a 64-bit Windows version of Meteor.

  • Step 3: Stop producing and supporting 32-bit versions of Meteor, and no longer support 32-bit architectures.

This PR

The sooner we can start letting people use Mongo 3.4.x, the better, so the code in this PR addresses Step 1 above. When using a 32-bit based OS, the Meteor Tool will use (and be limited to) Mongo 3.2.15. When using a 64-bit based OS, the Meteor Tool will use Mongo 3.4.9.

OS Architecture Meteor Tool Mongo Version
Linux 32-bit 3.2.15
Linux 64-bit 3.4.9
macOS 64-bit 3.4.9
Windows 32-bit 3.2.15
Windows 64-bit 3.4.9

A meteor build is produced and published for each OS/architecture combination in the table above (except Windows which is 32-bit only - more on this in a sec). This PR adjusts the Meteor Tool build process to use an appropriate version of Mongo based on a given OS/architecture combo.

This PR has been tested against all supported 32/64 bit OS', and is in a runnable state (if anyone wants to try things out). I've listed a few outstanding questions below that I'd love to get feedback on. Also, the tests will fail here until a new dev_bundle is generated.

Special Considerations for Windows:

Since we're still only building a 32-bit version of Meteor for Windows, this PR pulls both the 32 and 64 bit versions of Mongo into the 32-bit Windows Meteor Tool bundle. Then when running the tool, the appropriate version is used based on a quick check of the host OS's architecture (so even though we're still using a 32-bit version of Meteor on Windows, 64-bit Windows versions can use the latest 64-bit Mongo). This Windows approach is a stop-gap; when Step 2 above is addressed, it will no longer be necessary. Including both the 32 and 64 bit versions of Mongo in the Windows Meteor Tool bundle is fairly harmless however, and only increases the bundle size by approximately 40 megs.

Outstanding Questions:

  1. When using a 64-bit version of Linux with Mongo 3.4.9, depending on how the Linux environment has been setup, Mongo might output startup warnings when using Meteor's Mongo shell. For example:
MongoDB shell version v3.4.9                  
connecting to: mongodb://127.0.0.1:3001/meteor
MongoDB server version: 3.4.9
Server has startup warnings: 
2017-10-03T12:06:03.268+0000 I STORAGE  [initandlisten] 
2017-10-03T12:06:03.269+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-10-03T12:06:03.269+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-10-03T12:06:03.850+0000 I CONTROL  [initandlisten] 
2017-10-03T12:06:03.850+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-10-03T12:06:03.850+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-10-03T12:06:03.850+0000 I CONTROL  [initandlisten] 
2017-10-03T12:06:03.850+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-10-03T12:06:03.850+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-10-03T12:06:03.850+0000 I CONTROL  [initandlisten] 
2017-10-03T12:06:03.850+0000 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 3829 processes, 65536 files. Number of processes should be at least 32768 : 0.5 times number of files.
2017-10-03T12:06:03.850+0000 I CONTROL  [initandlisten] 

While these warnings are valid, they're (mostly) intended for production environments. Should we consider suppressing these warnings or leave them as is?

  1. When using a 3.2.x 32-bit version of Mongo (Linux and Windows) with the Meteor Tool, the Mongo shell will automatically dump a deprecation notice like:
MongoDB shell version: 3.2.15                 
connecting to: 127.0.0.1:3001/meteor
Server has startup warnings: 
2017-10-03T12:09:15.189+0000 I CONTROL  [initandlisten] 
2017-10-03T12:09:15.189+0000 I CONTROL  [initandlisten] ** WARNING: This 32-bit MongoDB binary is deprecated
2017-10-03T12:09:15.189+0000 I CONTROL  [initandlisten] 
2017-10-03T12:09:15.189+0000 I CONTROL  [initandlisten] 
2017-10-03T12:09:15.189+0000 I CONTROL  [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
2017-10-03T12:09:15.189+0000 I CONTROL  [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
2017-10-03T12:09:15.189+0000 I CONTROL  [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
2017-10-03T12:09:15.189+0000 I CONTROL  [initandlisten] **       See http://dochub.mongodb.org/core/32bit
2017-10-03T12:09:15.190+0000 I CONTROL  [initandlisten] 

Should we leave this as is or make it more user friendly?

Fixes meteor/meteor-feature-requests#129.

hwillson added 2 commits Oct 4, 2017
These changes introduce dual Mongo support into the Meteor
Tool. 32-bit Mongo (3.2.15) will be used by Meteor when the
Tool is run on a 32-bit OS (32-bit Linux and Windows). 64-bit
Mongo (3.4.9) will be used when the Tool is run on a 64-bit
OS (64-bit Linux, Windows and macOS).

Fixes meteor/meteor-feature-requests#129.
To test #9173.
@hwillson
Copy link
Member Author

@hwillson hwillson commented Oct 4, 2017

This PR was just reviewed in our weekly triage meeting, and the new plan is to go straight to Step 2 - building a 64-bit version of Meteor for Windows ( 🎉 ). I'll keep this PR open for now, since we'll be re-using parts of it shortly (and thanks for building the dev_bundle @abernix!).

@zimme
Copy link
Contributor

@zimme zimme commented Oct 4, 2017

  1. I'm in favour of silencing these errors.
  2. I don't have a problem keeping the warnings here, people should be moving to 64-bit systems.
@benjamn benjamn added this to the Release 1.6 milestone Oct 7, 2017
abernix added a commit that referenced this pull request Oct 9, 2017
To test #9173.
abernix added a commit that referenced this pull request Oct 10, 2017
To test #9173.
abernix added a commit that referenced this pull request Oct 10, 2017
This commit reverts much of the work @hwillson had put in place for
#9173, which made it possible for 32-bit and 64-bit
Mongo versions to exist in harmony within the same dev-bundle.  That
hard work was not in vein though as it offered invaluable research.
Ultimately, this showed that a more aggressive approach would be ideal,
even if the proposed option would have worked great in the short-term.

In the wake of the news that Mongo would no longer be supporting 32-bit
versions, these changes are important so 32-bit users of Meteor can
continue to have a functioning Mongo binary in development, while still
allowing Meteor to ship newer Mongo (e.g. 3.4+) binaries for 64-bit
users.  This is particularly relevant for Windows users, who have
previously only had 32-bit Meteor builds and represented a majority of
"32-bit" development, despite the fact most of their hosts supported
64-bit.  During another time in Meteor's life, this made sense.

This commit takes improved functionality to the next level (and makes
the aforementioned changes obsolete) by introducing support for building
and shipping Meteor for Windows in a 64-bit flavor (in addition to
32-bit), which will hopefully solve a number of performance matters on
Windows by using binaries which are pre-compiled for 64-bit, rather than
forcing the Windows kernel to deal with 32-bit binaries.  While Windows
has shown it's quite capable of dealing with such a situation, it seems
more and more clear (given improvements in underlying dependencies) that
performance gains could be had by freeing Windows of its 32-bit work.

This commit also further perpetuates the "archinfo" plot-line with more
story (about Windows) and various spelling corrections.
@hwillson
Copy link
Member Author

@hwillson hwillson commented Oct 11, 2017

Closing for now (Windows 64-bit changes are being made elsewhere and the necessary Mongo 3.4.x changes will be included as needed).

@hwillson hwillson closed this Oct 11, 2017
abernix added a commit that referenced this pull request Oct 12, 2017
This commit reverts much of the work @hwillson had put in place for
#9173, which made it possible for 32-bit and 64-bit
Mongo versions to exist in harmony within the same dev-bundle.  That
hard work was not in vein though as it offered invaluable research.
Ultimately, this showed that a more aggressive approach would be ideal,
even if the proposed option would have worked great in the short-term.

In the wake of the news that Mongo would no longer be supporting 32-bit
versions, these changes are important so 32-bit users of Meteor can
continue to have a functioning Mongo binary in development, while still
allowing Meteor to ship newer Mongo (e.g. 3.4+) binaries for 64-bit
users.  This is particularly relevant for Windows users, who have
previously only had 32-bit Meteor builds and represented a majority of
"32-bit" development, despite the fact most of their hosts supported
64-bit.  During another time in Meteor's life, this made sense.

This commit takes improved functionality to the next level (and makes
the aforementioned changes obsolete) by introducing support for building
and shipping Meteor for Windows in a 64-bit flavor (in addition to
32-bit), which will hopefully solve a number of performance matters on
Windows by using binaries which are pre-compiled for 64-bit, rather than
forcing the Windows kernel to deal with 32-bit binaries.  While Windows
has shown it's quite capable of dealing with such a situation, it seems
more and more clear (given improvements in underlying dependencies) that
performance gains could be had by freeing Windows of its 32-bit work.

This commit also further perpetuates the "archinfo" plot-line with more
story (about Windows) and various spelling corrections.
abernix added a commit that referenced this pull request Oct 12, 2017
This commit reverts much of the work @hwillson had put in place for
#9173, which made it possible for 32-bit and 64-bit
Mongo versions to exist in harmony within the same dev-bundle.  That
hard work was not in vein though as it offered invaluable research.
Ultimately, this showed that a more aggressive approach would be ideal,
even if the proposed option would have worked great in the short-term.

In the wake of the news that Mongo would no longer be supporting 32-bit
versions, these changes are important so 32-bit users of Meteor can
continue to have a functioning Mongo binary in development, while still
allowing Meteor to ship newer Mongo (e.g. 3.4+) binaries for 64-bit
users.  This is particularly relevant for Windows users, who have
previously only had 32-bit Meteor builds and represented a majority of
"32-bit" development, despite the fact most of their hosts supported
64-bit.  During another time in Meteor's life, this made sense.

This commit takes improved functionality to the next level (and makes
the aforementioned changes obsolete) by introducing support for building
and shipping Meteor for Windows in a 64-bit flavor (in addition to
32-bit), which will hopefully solve a number of performance matters on
Windows by using binaries which are pre-compiled for 64-bit, rather than
forcing the Windows kernel to deal with 32-bit binaries.  While Windows
has shown it's quite capable of dealing with such a situation, it seems
more and more clear (given improvements in underlying dependencies) that
performance gains could be had by freeing Windows of its 32-bit work.

This commit also further perpetuates the "archinfo" plot-line with more
story (about Windows) and various spelling corrections.
@abernix abernix mentioned this pull request Oct 12, 2017
benjamn added a commit that referenced this pull request Oct 14, 2017
This commit reverts much of the work @hwillson had put in place for
#9173, which made it possible for 32-bit and 64-bit
Mongo versions to exist in harmony within the same dev-bundle.  That
hard work was not in vein though as it offered invaluable research.
Ultimately, this showed that a more aggressive approach would be ideal,
even if the proposed option would have worked great in the short-term.

In the wake of the news that Mongo would no longer be supporting 32-bit
versions, these changes are important so 32-bit users of Meteor can
continue to have a functioning Mongo binary in development, while still
allowing Meteor to ship newer Mongo (e.g. 3.4+) binaries for 64-bit
users.  This is particularly relevant for Windows users, who have
previously only had 32-bit Meteor builds and represented a majority of
"32-bit" development, despite the fact most of their hosts supported
64-bit.  During another time in Meteor's life, this made sense.

This commit takes improved functionality to the next level (and makes
the aforementioned changes obsolete) by introducing support for building
and shipping Meteor for Windows in a 64-bit flavor (in addition to
32-bit), which will hopefully solve a number of performance matters on
Windows by using binaries which are pre-compiled for 64-bit, rather than
forcing the Windows kernel to deal with 32-bit binaries.  While Windows
has shown it's quite capable of dealing with such a situation, it seems
more and more clear (given improvements in underlying dependencies) that
performance gains could be had by freeing Windows of its 32-bit work.

This commit also further perpetuates the "archinfo" plot-line with more
story (about Windows) and various spelling corrections.
@crapthings
Copy link

@crapthings crapthings commented Nov 13, 2017

will meteor 1.6.1 shipping mongo 3.4.x ?

@crapthings
Copy link

@crapthings crapthings commented Nov 13, 2017

mongodb@release/METEOR@1.6.1-beta.5 is still 3.2.x

MONGO_VERSION_64BIT=3.2.15

@benjamn
Copy link
Member

@benjamn benjamn commented Nov 17, 2017

@hwillson @abernix Do we have a new issue (or PR) for the post-1.6 Mongo 3.4 update? Either way, let's get it in the 1.6.1 milestone!

abernix added a commit that referenced this pull request Nov 20, 2017
Most of the work to prepare for this change was done through the
excellent work of @hwillson in #9173 which, after some
re-working to support the 64-bit architecture on Windows platforms,
landed in #9218, making this change as simple as bumping
the minor version number (and rebuilding the dev bundle).

From this point forward, and due to Mongo's discontinuation of 32-bit
support in newer versions of MongoDB, 64-bit platforms will, in
development, use newer versions of Mongo while 32-bit architectures
will remain at 3.2.x versions.

Of course, in production, apps are free to use whichever version of
Mongo they would like, provided that version is supported by the
Node Mongo driver and Meteor's Mongo data packages.  At this time
there is no target for when Meteor will stop supporting Mongo 3.2,
but developers are encouraged to take steps to upgrade their Mongo
deployments (through their database providers) to newer versions
since Mongo has set September 2018 as the "End-of-Life" for Mongo
3.2.x.  For more information on Mongo support cycles, see their
support documents at https://www.mongodb.com/support-policy.

Refs: #9173
Refs: #9218
@abernix
Copy link
Member

@abernix abernix commented Nov 20, 2017

I've opened #9396, which was incredibly easy thanks to previous work by @hwillson in this PR.

abernix added a commit that referenced this pull request Nov 21, 2017
Most of the work to prepare for this change was done through the
excellent work of @hwillson in #9173 which, after some
re-working to support the 64-bit architecture on Windows platforms,
landed in #9218, making this change as simple as bumping
the minor version number (and rebuilding the dev bundle).

From this point forward, and due to Mongo's discontinuation of 32-bit
support in newer versions of MongoDB, 64-bit platforms will, in
development, use newer versions of Mongo while 32-bit architectures
will remain at 3.2.x versions.

Of course, in production, apps are free to use whichever version of
Mongo they would like, provided that version is supported by the
Node Mongo driver and Meteor's Mongo data packages.  At this time
there is no target for when Meteor will stop supporting Mongo 3.2,
but developers are encouraged to take steps to upgrade their Mongo
deployments (through their database providers) to newer versions
since Mongo has set September 2018 as the "End-of-Life" for Mongo
3.2.x.  For more information on Mongo support cycles, see their
support documents at https://www.mongodb.com/support-policy.

Refs: #9173
Refs: #9218
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.