Update MongoDB to 4.0

[klaussner]

This PR updates MongoDB to version 4.0 and the driver to version 3.1.
Fixes meteor/meteor-feature-requests#269.

[klaussner]

The driver shows this warning by default:

DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.

[hwillson]

This is awesome @klaussner - thanks very much!

[hwillson]

@klaussner One small thing - it looks like 3.6.4 is still being used for meteor mongo?

[klaussner]

@hwillson Did you rebuild the dev bundle? This is the output I get from the mongo shell:

$ meteor-dev mongo
MongoDB shell version v4.0.0
connecting to: mongodb://127.0.0.1:3001/meteor
MongoDB server version: 4.0.0
Server has startup warnings:
2018-07-12T21:40:42.422+0200 I STORAGE  [initandlisten]
2018-07-12T21:40:42.422+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-07-12T21:40:42.422+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
meteor:PRIMARY>

[hwillson]

Hi @klaussner - I did rebuild the dev_bundle, but when I just tried again I'm now seeing bizarre pipelineOperator errors whenever I try to run the new build. I tested this out on a new laptop (with this as the first Meteor test), so I'm going to assume something is up on my end. The code changes look good, so my earlier 👍 still stands. I'll post back shortly after I reconfigure things, but if you're seeing the proper Mongo shell, then I'm happy. Thanks again for working on this!

[hwillson]

Quick update relating to #10058 (comment) - the pipelineOperator issue was fixed in benjamn/reify@9985a49, and will be addressed in devel shortly (which will make testing this PR much easier).

[hwillson]

After running with the new reify version, I now see:

➜  pr-10058 /Users/hwillson/Documents/git/meteor/forks/klaussner/meteor/meteor mongo
MongoDB shell version v4.0.0                  
connecting to: mongodb://127.0.0.1:3001/meteor
MongoDB server version: 4.0.0
---
Enable MongoDB's free cloud-based monitoring service to collect and display
metrics about your deployment (disk utilization, CPU, operation statistics,
etc).

The monitoring data will be available on a MongoDB website with a unique
URL created for you. Anyone you share the URL with will also be able to
view this page. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command:
db.enableFreeMonitoring()
---

meteor:PRIMARY> 

Should we suppress Mongo's free monitoring message (is that dynamically pulled in?), or leave it as is? Either way, it's working - thanks @klaussner!

[klaussner]

Interesting, I didn't get the free monitoring message, just the filesystem warning. 🤷‍♂️ The only way I found to suppress them is to pass the --quiet option to the mongo shell (here) but then the version info and URI also disappear:

$ meteor-dev mongo
meteor:PRIMARY>

On the other hand, they can easily be found using the shell, so I think I'm in favor of using the --quiet option:

$ meteor-dev mongo
meteor:PRIMARY> version()
4.0.0
meteor:PRIMARY> db.serverCmdLineOpts().parsed.net
{ "bindIp" : "127.0.0.1", "port" : 3001 }
meteor:PRIMARY>

[hwillson]

I like how nice and clean things look using --quiet so I'm all for it. Mongo newcomers might not know how to get Mongo version information, but it is pretty easy to find out. So my vote is for --quiet; @benjamn @abernix what do you guys think?

[hwillson]

Hi @klaussner - 👍 on the --quiet flag. If you don't mind making that change, we'll get this merged (and if you're busy just let me know and I'll make the necessary changes). Thanks!

[klaussner]

Thanks, @hwillson. I've added the --quiet flag and rebased onto the current devel. 🙂

[benjamn]

I've rebased these changes onto release-1.7.1 and built a new dev bundle (8.11.3.13): a875ac5

I think there may still be some test failures, judging from local meteor self-test attempts. Whenever you have a chance to look into that, just rebase against release-1.7.1, change the target branch of this PR, and add any additional commits as you see fit.

Thanks for working on this @klaussner!

[benjamn]

I went digging for another way to disable the free monitoring advertisement, and I found this command-line option: --enableFreeMonitoring off

However, when I add that option to the mongod arguments, the process crashes with a segmentation fault!

At least, that's what happens on OSX. On Linux, I get

Error parsing command line: unrecognised option '--enableFreeMonitoring'
try '/home/ben/meteor/dev_bundle/mongodb/bin/mongod --help' for more information

I wrote up these findings in their JIRA issue pertaining to the problem: https://jira.mongodb.org/browse/SERVER-36474

It's hard to know how worried we should be about the general stability of Mongo 4.0.0. Should we be concerned about merely silencing (rather than disabling) this poorly-tested feature?

[mitar]

@klaussner Have you checked how the new transactions feature interact with Meteor's pub/sub and oplog tailing? We should make sure that if you do a transaction and modify data, that this is not leaked through oplog to pub/sub. Probably pub/sub should push updates to the client only after the transaction commits.

[klaussner]

@mitar I haven't tried it but I assume that transactions don't work with pub/sub at the moment. Each transaction has its own oplog entry, which includes a list of operations (inserts, updates, etc.) that were committed successfully but Meteor doesn't handle this type of oplog entry yet, I think.

[mitar]

I think oplog entries just have a transaction ID associated with it. What I worry is that if Meteor oplog tailing ignores that entry, it might update based on some operations happening inside the transaction, which should not yet be visible outside. Or are you saying that operations inside transactions are not send over the oplog until committed? I do not know details here though. Maybe you are right. I just see a potential issue here.

[klaussner]

Exactly, only operations of successful transactions will appear on the oplog. I opened an issue with more details: #10195.