Support OpLog

[ferretdb-bot]

We want to support a subset MongoDB's OpLog that is enough for applications and frameworks tailing it, such as Meteor.
It is not a goal (yet) to make it good enough for replication.

See:

• https://github.com/meteor/meteor/blob/devel/packages/mongo/oplog_observe_driver.js
• https://github.com/meteor/meteor/blob/devel/packages/mongo/oplog_tailing.js
• https://github.com/meteor/meteor/blob/devel/packages/mongo/oplog_v2_converter.js

We should support the "old" format, with full $set for updates, not the new format with diff.

Tasks

Beta Give feedback

1. Fix tests for MongoDB with enabled replica set #3310
   code/chore +1
   
2. Add integration tests for OpLog i/u/d #3861
   code/chore +1
   
3. Insert i/u/d documents into OpLog #3556
   area/oplog code/chore +2
4. Insert no-op OpLog documents #3500
   area/oplog code/chore +2
   
5. Add tests for OpLog tailing #3868
   code/chore not ready +2
   

[ocdtrekkie]

Hello! We have been sort of following this project with interest/excitement for a while, and are potentially interested in using FerretDB with a Meteor app to escape a presently extremely old version of MongoDB as a dependency.

Do you have any ballpark how far out Meteor compatibility for FerretDB might be in your present plans?

[mitar]

Hm, in this blog post it says:

We are testing FerretDB against real-world applications, like FastNetMon, the Enmeshed Connector, BigBlueButton, Strapi, or frameworks such as MeteorJS, among others.

So what does this mean? Is oplog supported or not? Because from reading the post I understood that Meteor (and oplog) support is already available.

[AlekSi]

Meteor fallbacks to polling when OpLog is not available. It is not suitable for many production deployments, but sometimes it is good enough. In that case, Meteor applications could work with FerretDB, and indeed some of them work.

[DblK]

I just found out another library in Go similar to FerretDb and with already support for Oplog, it's called lungo.
It's MIT licensed so you might be able to take some inspiration there ;)

[radekmie]

I'm not sure how advanced this effort is, but I wanted to cross-post meteor/meteor#12150 (reply in thread) here:

Update from my end: as soon as FerretDB supports Change Streams, Meteor can stay fully reactive with cultofcoders:redis-oplog and my new tool changestream-to-redis.

In other words: #175 is enough for Meteor apps, as long as they want to work with cultofcoders:redis-oplog.

[AlekSi]

The first version with OpLog support is out! See https://github.com/FerretDB/FerretDB/releases/tag/v1.18.0