Skip to content

Update MongoDB to 4.2 #10723

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 13, 2020
Merged

Update MongoDB to 4.2 #10723

merged 5 commits into from
Jan 13, 2020

Conversation

klaussner
Copy link
Contributor

@klaussner klaussner commented Oct 3, 2019

To do:

@CLAassistant
Copy link

CLAassistant commented Oct 3, 2019

CLA assistant check
All committers have signed the CLA.

Copy link
Contributor

@benjamn benjamn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for getting this started. Those to-dos are going to be the hard part, I suspect.

@rj-david
Copy link
Contributor

Just curious why meteor did not go the docker route for mongodb? More work for devs?

@klaussner
Copy link
Contributor Author

I've updated all URLs to the latest versions. For now, the custom Linux build is downloaded from the GitHub releases of my mongodb-builder repo.

@klaussner klaussner marked this pull request as ready for review October 20, 2019 11:30
@benjamn benjamn added this to the Release 1.8.3 milestone Oct 24, 2019
@klaussner
Copy link
Contributor Author

Regarding the custom MongoDB build: Should I transfer the mongodb-builder repo to the Meteor organization? Then the Meteor team could build the binaries and host them on GitHub when a new MongoDB version is released. Or do you prefer a different way to build and host the files? Just let me know and I'll update this PR with the new URLs. 🙂

@filipenevola
Copy link
Contributor

Sure @klaussner , go ahead and transfer the repo to Meteor org, I'll provide write access to you also!

@filipenevola
Copy link
Contributor

mongodb-builder repo is now under meteor org https://github.com/meteor/mongodb-builder

@benjamn benjamn modified the milestones: Release 1.8.4, Release 1.9 Dec 18, 2019
@benjamn benjamn changed the base branch from devel to release-1.9 December 18, 2019 21:52
@benjamn benjamn force-pushed the mongodb-4.2 branch 3 times, most recently from ffe172a to 87f5675 Compare December 18, 2019 21:59
Copy link
Contributor

@benjamn benjamn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've rebuilt the dev bundle with Mongo 4.2, and I will merge these changes into release-1.9 assuming the tests pass. Thanks for doing all the real work @klaussner!

documentation: null
});

Npm.depends({
mongodb: "3.2.7"
mongodb: "3.4.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: I took the liberty of updating the mongodb npm package to the latest version (@klaussner had used 3.3.2 in the original version of this PR). Maybe that was not a good idea?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good idea 😉 I believe @klaussner was going to do the same.

@klaussner
Copy link
Contributor Author

I haven't figured out yet why the "mongo with multiple --port numbers" test doesn't pass. MongoDB outputs file locking errors when the port is changed. 🤔

2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] MongoDB starting : pid=19516 port=4001 dbpath=<APP>/.meteor/local/db 64-bit host=arch
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] db version v4.2.1
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] git version: edf6d45851c0b9ee15548f0f847df141764a317e
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] allocator: tcmalloc
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] modules: none
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] build environment:
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten]     distarch: x86_64
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten]     target_arch: x86_64
2019-12-21T14:04:36.151+0100 I  CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 4001 }, replication: { oplogSizeMB: 8, replSet: "meteor" }, security: { authorization: "disabled" }, storage: { dbPath: "<APP>/.meteor/local/db" } }
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] Detected data files in <APP>/.meteor/local/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] 
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-12-21T14:04:36.151+0100 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7471M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (11) [1576933476:609245][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable Raw: [1576933476:609245][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (16) [1576933476:609286][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy Raw: [1576933476:609286][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (11) [1576933476:609381][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable Raw: [1576933476:609381][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (16) [1576933476:609391][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy Raw: [1576933476:609391][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (11) [1576933476:609517][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable Raw: [1576933476:609517][19516:0x7f83d4b3a340], wiredtiger_open: __posix_file_lock, 391: <APP>/.meteor/local/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable
2019-12-21T14:04:36.609+0100 E  STORAGE  [initandlisten] WiredTiger error (16) [1576933476:609525][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy Raw: [1576933476:609525][19516:0x7f83d4b3a340], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy

@klaussner klaussner force-pushed the mongodb-4.2 branch 2 times, most recently from 0ee6c5b to 6843b25 Compare December 27, 2019 19:40
@klaussner
Copy link
Contributor Author

I found out what the problem was. The test runs Meteor multiple times in a row but mongod didn't shut down before the next process attempted to acquire the lock. Starting with version 4.0.8/4.1.10, MongoDB performs a step down procedure if the primary receives a SIGTERM signal (https://jira.mongodb.org/browse/SERVER-38994). During this procedure, the process doesn't shut down for up to ten seconds until a secondary becomes the new primary. Since Meteor starts a single-node replica set, this is unnecessary because there are no secondaries. Setting the waitForStepDownOnNonCommandShutdown parameter to false when starting MongoDB disables the step down. This will be the default for single-node replica sets in MongoDB 4.3 (relevant commit: https://git.io/JeNkT), so the parameter can be removed in the future. Hopefully, the test passes now. 🤞

I also removed the useNewUrlParser option in 83e087b878283bb95599e9b200aa9f3e09ce0f11 because it defaults to true since mongodb 3.3.3.

@@ -3,12 +3,12 @@

Package.describe({
summary: "Wrapper around the mongo npm package",
version: "3.3.0-rc190.0",
version: "3.4.0-rc190.1",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the right version number?

@klaussner
Copy link
Contributor Author

The Travis CI build fails because the custom mongod binary links to a newer version of libstdc++ than is installed in the Travis CI environment (which runs Ubuntu 16.04). I think this can be solved by running the custom build with an older Docker base image or by statically linking libstdc++. 🤔

@klaussner
Copy link
Contributor Author

I changed the build script to link libstdc++ statically and created a new release. The link to the archive is the same as before but we need a new dev bundle. @benjamn 👀

@benjamn benjamn modified the milestones: Release 1.9.x, Release 1.10 Jan 13, 2020
@benjamn benjamn changed the base branch from release-1.9 to devel January 13, 2020 21:19
@benjamn benjamn merged commit 3bc5352 into meteor:devel Jan 13, 2020
@benjamn benjamn mentioned this pull request Jan 14, 2020
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants