Exception in flushing DDP buffered writes: MinimongoError: Key $v must not start with '$' - only when using oplog

[Shelagh-Lewins]

I have deployed a Meteor app through Phusion Passenger / Nginx, and see this error in the browser console when I update a collection:

Exception in flushing DDP buffered writes: MinimongoError: Key $v must not start with '$'

The error only occurs if the app is configured to use the oplog. "$v" is NOT the name of the field I'm updating, and as far as I can tell does not exist in my code.

OS: RedHat 8
Nginx: v1.14.1
Node.js: v14.18.2
Meteor: v2.5.1
Phusion Passenger: v6.0.12
MongoDB shell version v5.0.5

To enable the oplog, I first added this to the end of the Mongo configuration file:

/etc/mongod.conf

replication:
  replSetName: rs0
  oplogSizeMB: 100

Then added the hostname as an alias by first running:

hostname

This returns the hostname.

Add this to the hosts file:

/etc/hosts

127.0.0.1 myhostname

Note: /etc/hosts is managed by Puppet, as shown by the header text

# HEADER: This file was autogenerated at xxx
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.

However, I have checked and my entry in the file is still present when the issue occurs.

Then restart mongodb:

sudo service mongod stop
sudo service mongod start

Run the Mongo shell:

mongo

And in the mongo shell:

use local
rs.initiate()

The Mongo log shows this text which makes me think the replica set is working:

,"replication":{"oplogSizeMB":100,"replSetName":"rs0"}

and

"Replication config state is Steady, starting reconfig"

Add an entry in Nginx / Passenger config to use oplog:

passenger_env_var MONGO_OPLOG_URL mongodb://localhost:27017/local;

and restart Nginx:

sudo service nginx restart

If I comment out this line and restart Nginx, the error does not occur. When I restore the MONGO_OPLOG_URL and restart Nginx, the error happens.

Here's the code I'm running in the server in Meteor to update the collection:

const newValues = {"description":"An updated description"}
MyCollection.update({ documentId }, { '$set': newValues });

The only other reference I've seen to this bug was on the Minimongo github: Exception in flushing DDP buffered writes: MinimongoError: Key $v must not start with '$', but the submitter was advised that Meteor MiniMongo is a standalone and questions should be directed to the Meteor GitHub.

The likeliest thing seems to be that I have missed some step in configuring / enabling the oplog, but it is also possible there's a bug in Meteor's use of MiniMongo. I found very few instructions for this and I'm using the steps which worked for Meteor 1.8. I can't find anything online that suggests how I might debug this or find out what is the issue. I'll be very grateful for help / suggestions!

[vitorflores]

Hi, could you reproduce it locally using mongo embedded? This way we can help you better, checking if the problem is with your config file or with MiniMongo.

[Shelagh-Lewins]

Hi, thank you, but no, I cannot reproduce it locally, only on the remote server. This seems strange in itself! On the remote server, everything else works fine; it is only with Meteor configured to use the oplog that there is a problem. I cannot see any packages that would be causing this issue, and can't think of a way to investigate this further.

[shelagh-lewins-ucl]

After downgrading my setup, I've worked out how to reproduce the issue.

1. run Mongodb 5.0
2. set up a replica set, and hence the oplog, in mongo
3. configure Meteor to use the oplog by setting the passenger_env_var MONGO_OPLOG_URL

Now I know to look for Mongo 5.0 issues, I found a reference to this exact problem:

https://forums.meteor.com/t/mongodb-5-0-is-out/56266/3

The post suggests that the problem can be worked around using:

db.adminCommand( { setFeatureCompatibilityVersion: “4.4” } )

but for production I'm guessing I should stick to Mongo 4.4.

I see there is some discussion about Meteor compatibility with Mongo 5.0 here:

#11749

[renanccastro]

Hi @shelagh-lewins-ucl , yes, we currently doesn't support MongoDB 5.0.

We are actively working on it, so you should expect news soon. In the meanwhile, it's safer to use 4.4

[renanccastro]

I will close this in the meanwhile, ok? If you need further assistance after the 5.0 PR lands, we can reopen this issue. Thanks!

[kakadais]

This is not about Server Mongo but for MiniMongo only.

The data {$xxx: data} is saved on server-mongo but when it is published and subscribed on the client,
occurs an error below.

common.js:1087 Uncaught MinimongoError: Key $a must not start with '$'
    at MinimongoError (common.js:1087:17)
    at assertIsValidFieldName (local_collection.js:1922:11)
    at Object.<anonymous> (local_collection.js:1913:7)
    at JSON.stringify (<anonymous>)
    at assertHasValidFieldNames (local_collection.js:1912:10)
    at LocalCollection.insert (local_collection.js:102:5)
    at Object.update (collection.js:239:28)
    at store.<computed> [as update] (livedata_connection.js:312:38)
    at livedata_connection.js:1340:19
    at Array.forEach (<anonymous>)

[kakadais]

Can you re-open this issue? @renanccastro