Skip to content
This repository has been archived by the owner. It is now read-only.

Upgrade to MongoDB 4.2 support #361

Closed
benlavalley opened this issue Aug 16, 2019 · 13 comments
Closed

Upgrade to MongoDB 4.2 support #361

benlavalley opened this issue Aug 16, 2019 · 13 comments

Comments

@benlavalley
Copy link

@benlavalley benlavalley commented Aug 16, 2019

Now that MongoDB 4.2 is released, Meteor should be updated to support it.

It introduces a few new key features, namely faster replica set failover handling and initial sync, lower cpu / higher compression algorithm "Zstandard" / Zstd, wildcard support for indexes (huge), and online index builds (also huge).

The NodeJs driver for Mongodb 4.2 was released 3 days ago, so hopefully we can see it included in Meteor soon.

@benjamn <3 :)

@mitar
Copy link

@mitar mitar commented Aug 16, 2019

Is there anything necessary on Meteor side, except for the driver update?

@benlavalley
Copy link
Author

@benlavalley benlavalley commented Aug 16, 2019

Hopefully not. I looked through and see a lot of refactoring, rewrites and some new features, but didn't spot any breaking API changes (though there is a new mechanism for topology selection introduced in 3.2, that will be mandatory at some point in the future in 4.x). I was debating trying it out locally if I can, if only for the wildcard index support... I was just about to re-factor indexes on a massive collection but will hold off until I can use 4.2.

Going to try the trick to edit the npm-mongo wrapper package locally with a development database and see how it goes.

@benlavalley
Copy link
Author

@benlavalley benlavalley commented Aug 20, 2019

Few days now using the new mongo 3.3.0 driver with 4.0 and 4.2 mongo DBs - so far so good. I have to hold off on updating my Mongo featureCompatibilityVersion to 4.2 until Mongo's OpsManager backup support catches up with 4.2.

For anyone curious, it was just a matter of cloning the Meteor npm-mongo wrapper package into /packages, and updating package.js for the mongodb 3.3.0. You can verify the driver update with the MongoInternals.NpmModules.mongodb.version command.

@StorytellerCZ
Copy link

@StorytellerCZ StorytellerCZ commented Aug 20, 2019

@benlavalley Can you submit a PR with your update. It is a quick improvement, but the first step to add additional features that have been requested by other feature requests.

@benlavalley
Copy link
Author

@benlavalley benlavalley commented Aug 30, 2019

@StorytellerCZ Will update my production instances to feature compatibility of 4.2 (one which Meteor monitors the oplog for, another without oplog monitoring enabled) and provided all goes well I'll get it submitted.

I also browsed the Meteor source tree to see if I could figure out getting Meteor's default development database updated to 4.2 as well.

Lots of reasons to look at MongoDB 4.2 especially for production apps.

@StorytellerCZ
Copy link

@StorytellerCZ StorytellerCZ commented Aug 30, 2019

I took a look at the code quickly and I don't see anything that would create an issue (but testing never hurts).

@klaussner
Copy link

@klaussner klaussner commented Sep 11, 2019

Updating the bundled MongoDB binaries (mongod and mongo) is going to be more difficult this time because MongoDB stopped releasing generic Linux builds, which Meteor currently uses. The advantage of these builds was that they run on any Linux distribution. Starting with MongoDB 4.2, you can only download builds that work on specific versions of Linux distributions. For example, the build for Ubuntu 16.04 won't work on Ubuntu 18.04 and vice versa because the operating systems ship with different OpenSSL versions.

I've downloaded some of the MongoDB builds and compared their dependencies:

Last available generic build (MongoDB 4.0.12):

libresolv.so.2
libdl.so.2
librt.so.1
libm.so.6
libgcc_s.so.1
libpthread.so.0
libc.so.6
ld-linux-x86-64.so.2

Debian 9, Ubuntu 18.04 (MongoDB 4.2.0):

+ libcurl.so.4
libresolv.so.2
+ libcrypto.so.1.1
+ libssl.so.1.1
libdl.so.2
librt.so.1
libm.so.6
libgcc_s.so.1
libpthread.so.0
libc.so.6
ld-linux-x86-64.so.2

Ubuntu 16.04, SUSE 12 (MongoDB 4.2.0):

+ libcurl.so.4
libresolv.so.2
+ libcrypto.so.1.0.0
+ libssl.so.1.0.0
libdl.so.2
librt.so.1
libm.so.6
libgcc_s.so.1
libpthread.so.0
libc.so.6
ld-linux-x86-64.so.2

As you can see in the diffs, the builds link to different versions of libcrypto and libssl (both are part of OpenSSL). I think we have three options to solve this problem:

  1. Stop bundling MongoDB with Meteor and let developers install the required version manually.
  2. Bundle a MongoDB build that links to the most recent versions of its dependencies and make it the responsibility of developers to install the necessary libraries (even if their operating systems don't provide them in their official repositories).
  3. Figure out a way to build MongoDB from source without SSL support, which would result in binaries that are similar to the generic builds that Meteor currently uses.

I think (1) and (2) would impact developer experience too much and (3) would require a fair amount of additional work. Any suggestions? 😅

@mitar
Copy link

@mitar mitar commented Sep 11, 2019

Why not simply bundle dependencies as well and when starting mongodb process during development, we configure things so that it looks in local directory for those dependencies instead of system directory, by setting LD_LIBRARY_PATH.

@klaussner
Copy link

@klaussner klaussner commented Sep 13, 2019

That could work for the OpenSSL libraries but I'm not sure about libcurl because it depends on other dynamic libraries that we would probably also have to include (https://packages.debian.org/de/sid/libcurl4).

@mitar
Copy link

@mitar mitar commented Sep 13, 2019

Sure. You might have to do it recursively. But there is a finite amount of them. :-)

@klaussner
Copy link

@klaussner klaussner commented Sep 22, 2019

I tried to build MongoDB from source and it was easier than expected. This Docker image contains all build dependencies and scripts: https://github.com/klaussner/mongodb-builder. The mongod and mongo binaries for MongoDB 4.2.0 can be found in the release tab.

@benjamn
Copy link
Member

@benjamn benjamn commented Sep 23, 2019

@klaussner Thanks so much for doing this investigation! I think building from source is a reasonable path, especially since this is just the development server (so SSL doesn't matter as much), and assuming the Mongo build is decoupled from the dev bundle build, so we don't have to rebuild Mongo every time we rebuild the dev bundle. These new Linux-distribution-specific builds/URLs really are a pain!

@afrokick
Copy link

@afrokick afrokick commented May 4, 2020

Should be closed @filipenevola ?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants