Update MongoDB to 4.2 #10723

klaussner · 2019-10-03T16:21:57Z

To do:

Download MongoDB 4.2 in dev bundle scripts.
Use custom build for Linux (Upgrade to MongoDB 4.2 support meteor-feature-requests#361).
Test on macOS, Windows, and 32-bit Linux

CLAassistant · 2019-10-03T16:22:40Z

All committers have signed the CLA.

benjamn

Thanks for getting this started. Those to-dos are going to be the hard part, I suspect.

packages/npm-mongo/package.js

rj-david · 2019-10-11T01:00:02Z

Just curious why meteor did not go the docker route for mongodb? More work for devs?

klaussner · 2019-10-20T11:30:21Z

I've updated all URLs to the latest versions. For now, the custom Linux build is downloaded from the GitHub releases of my mongodb-builder repo.

klaussner · 2019-11-22T21:22:15Z

Regarding the custom MongoDB build: Should I transfer the mongodb-builder repo to the Meteor organization? Then the Meteor team could build the binaries and host them on GitHub when a new MongoDB version is released. Or do you prefer a different way to build and host the files? Just let me know and I'll update this PR with the new URLs. 🙂

filipenevola · 2019-12-11T13:05:49Z

Sure @klaussner , go ahead and transfer the repo to Meteor org, I'll provide write access to you also!

filipenevola · 2019-12-16T13:07:45Z

mongodb-builder repo is now under meteor org https://github.com/meteor/mongodb-builder

benjamn

I've rebuilt the dev bundle with Mongo 4.2, and I will merge these changes into release-1.9 assuming the tests pass. Thanks for doing all the real work @klaussner!

benjamn · 2019-12-18T23:47:34Z

packages/npm-mongo/package.js

  documentation: null
 });

 Npm.depends({
-  mongodb: "3.2.7"
+  mongodb: "3.4.0"


Note: I took the liberty of updating the mongodb npm package to the latest version (@klaussner had used 3.3.2 in the original version of this PR). Maybe that was not a good idea?

This is a good idea 😉 I believe @klaussner was going to do the same.

klaussner · 2019-12-21T13:32:25Z

I haven't figured out yet why the "mongo with multiple --port numbers" test doesn't pass. MongoDB outputs file locking errors when the port is changed. 🤔

2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] MongoDB starting : pid=19516 port=4001 dbpath=<APP>/.meteor/local/db 64-bit host=arch
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] db version v4.2.1
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] git version: edf6d45851c0b9ee15548f0f847df141764a317e
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] allocator: tcmalloc
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] modules: none
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] build environment:
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten]     distarch: x86_64
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten]     target_arch: x86_64
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 4001 }, replication: { oplogSizeMB: 8, replSet: "meteor" }, security: { authorization: "disabled" }, storage: { dbPath: "<APP>/.meteor/local/db" } }
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] Detected data files in <APP>/.meteor/local/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] 
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7471M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (11) [1576933476:609245][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable Raw: [1576933476:609245][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (16) [1576933476:609286][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy Raw: [1576933476:609286][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (11) [1576933476:609381][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable Raw: [1576933476:609381][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (16) [1576933476:609391][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy Raw: [1576933476:609391][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (11) [1576933476:609517][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable Raw: [1576933476:609517][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (16) [1576933476:609525][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy Raw: [1576933476:609525][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy

klaussner · 2019-12-27T19:44:07Z

I found out what the problem was. The test runs Meteor multiple times in a row but mongod didn't shut down before the next process attempted to acquire the lock. Starting with version 4.0.8/4.1.10, MongoDB performs a step down procedure if the primary receives a SIGTERM signal (https://jira.mongodb.org/browse/SERVER-38994). During this procedure, the process doesn't shut down for up to ten seconds until a secondary becomes the new primary. Since Meteor starts a single-node replica set, this is unnecessary because there are no secondaries. Setting the waitForStepDownOnNonCommandShutdown parameter to false when starting MongoDB disables the step down. This will be the default for single-node replica sets in MongoDB 4.3 (relevant commit: https://git.io/JeNkT), so the parameter can be removed in the future. Hopefully, the test passes now. 🤞

I also removed the useNewUrlParser option in 83e087b878283bb95599e9b200aa9f3e09ce0f11 because it defaults to true since mongodb 3.3.3.

klaussner · 2019-12-27T19:46:07Z

packages/npm-mongo/package.js

@@ -3,12 +3,12 @@

 Package.describe({
  summary: "Wrapper around the mongo npm package",
-  version: "3.3.0-rc190.0",
+  version: "3.4.0-rc190.1",


Is this the right version number?

klaussner · 2019-12-28T21:21:07Z

The Travis CI build fails because the custom mongod binary links to a newer version of libstdc++ than is installed in the Travis CI environment (which runs Ubuntu 16.04). I think this can be solved by running the custom build with an older Docker base image or by statically linking libstdc++. 🤔

klaussner · 2019-12-29T17:29:29Z

I changed the build script to link libstdc++ statically and created a new release. The link to the archive is the same as before but we need a new dev bundle. @benjamn 👀

benjamn reviewed Oct 3, 2019

View reviewed changes

packages/npm-mongo/package.js Outdated Show resolved Hide resolved

lynchem mentioned this pull request Oct 8, 2019

Allow use of useUnifiedTopology in Mongo Driver meteor/meteor-feature-requests#368

Closed

klaussner force-pushed the mongodb-4.2 branch from e9588a0 to ebf6bf1 Compare October 9, 2019 17:53

klaussner force-pushed the mongodb-4.2 branch from 7f82eff to ee9c7b1 Compare October 20, 2019 11:01

klaussner marked this pull request as ready for review October 20, 2019 11:30

benjamn added this to the Release 1.8.3 milestone Oct 24, 2019

benjamn added the Project:Mongo Driver label Oct 24, 2019

klaussner force-pushed the mongodb-4.2 branch from ee9c7b1 to d7aa66f Compare November 22, 2019 21:20

benjamn modified the milestones: Release 1.8.4, Release 1.9 Dec 18, 2019

benjamn changed the base branch from devel to release-1.9 December 18, 2019 21:52

benjamn force-pushed the mongodb-4.2 branch 3 times, most recently from ffe172a to 87f5675 Compare December 18, 2019 21:59

benjamn approved these changes Dec 18, 2019

View reviewed changes

benjamn reviewed Dec 18, 2019

View reviewed changes

klaussner force-pushed the mongodb-4.2 branch 2 times, most recently from 0ee6c5b to 6843b25 Compare December 27, 2019 19:40

klaussner commented Dec 27, 2019

View reviewed changes

filipenevola mentioned this pull request Jan 10, 2020

[1.8] ReplicaSet can't begin with number #10362

Closed

benjamn modified the milestones: Release 1.9.x, Release 1.10 Jan 13, 2020

klaussner and others added 4 commits January 13, 2020 16:02

Update MongoDB driver to 3.4.0.

282b16e

Update download URLs for MongoDB 4.2.1.

1a016f5

Bump $BUNDLE_VERSION to 12.14.0.5 before rebuilding dev bundle.

ae502d3

Don't wait for mongod to step down on shutdown.

e967e74

benjamn force-pushed the mongodb-4.2 branch from 6843b25 to e967e74 Compare January 13, 2020 21:19

benjamn changed the base branch from release-1.9 to devel January 13, 2020 21:19

Mention MongoDB 4.2 update in History.md.

6cc98ca

benjamn merged commit 3bc5352 into meteor:devel Jan 13, 2020

benjamn mentioned this pull request Jan 14, 2020

Release 1.10 #10861

Merged

5 tasks

Update MongoDB to 4.2 #10723

Update MongoDB to 4.2 #10723

klaussner commented Oct 3, 2019 •

edited

CLAassistant commented Oct 3, 2019 •

edited

benjamn left a comment

rj-david commented Oct 11, 2019

klaussner commented Oct 20, 2019

klaussner commented Nov 22, 2019

filipenevola commented Dec 11, 2019

filipenevola commented Dec 16, 2019

benjamn left a comment

benjamn Dec 18, 2019

filipenevola Dec 19, 2019

klaussner commented Dec 21, 2019

klaussner commented Dec 27, 2019

klaussner Dec 27, 2019

klaussner commented Dec 28, 2019

klaussner commented Dec 29, 2019

Update MongoDB to 4.2 #10723

Update MongoDB to 4.2 #10723

Conversation

klaussner commented Oct 3, 2019 • edited

CLAassistant commented Oct 3, 2019 • edited

benjamn left a comment

Choose a reason for hiding this comment

rj-david commented Oct 11, 2019

klaussner commented Oct 20, 2019

klaussner commented Nov 22, 2019

filipenevola commented Dec 11, 2019

filipenevola commented Dec 16, 2019

benjamn left a comment

Choose a reason for hiding this comment

benjamn Dec 18, 2019

Choose a reason for hiding this comment

filipenevola Dec 19, 2019

Choose a reason for hiding this comment

klaussner commented Dec 21, 2019

klaussner commented Dec 27, 2019

klaussner Dec 27, 2019

Choose a reason for hiding this comment

klaussner commented Dec 28, 2019

klaussner commented Dec 29, 2019

klaussner commented Oct 3, 2019 •

edited

CLAassistant commented Oct 3, 2019 •

edited