BSON field 'insert.bypassEmptyTsReplacement' is an unknown field

[AlekSi]

Purpose of the feature.

Current versions of mongoexport/mongoimport and mongodump/mongorestore set bypassEmptyTsReplacement parameter that DocumentDB does not support. That prevents them from working.

Describe the solution you'd like

This parameter is implemented.
It does not look like this parameter can be safely ignored.

Describe alternatives you've considered

I considered removing that parameter on the FerretDB side, but it is not safe.

Additional context

https://jira.mongodb.org/browse/SERVER-88750

[shuaitian-git]

We're recently optimizing insert/update while trying to implement bulkWrite, will investigate this along with that and see how to support this param.

[kbaegis]

Very interested in this feature. Currently breaking mongorestore and migration to documentdb.

(UnknownBsonField) BSON field 'insert.bypassEmptyTsReplacement' is an unknown field

[duanpei-ms]

Can you give more context? We are considering whether we can just ignore this field if mongorestore and migration don't rely on the behavior of converting timestamp(0,0) to current timestamp.

[AlekSi]

The fact that DocumentDB seems to insert timestamp(0,0) as timestamp(0,0) is a bug by itself – MongoDB inserts the current timestamp.

More context is available in MongoDB's Jira ticket. In short, the current version of backup/restore tools just exit with an error due to an unsupported parameter.

[abinav2307]

@shuaitian-git - can this be created into a Task and added to the DocumentDB Roadmap project?

[lichoil]

@abinav2307 ,we have internal work item assigned to @duanpei-ms who is working on this.

[lichoil]

@kbaegis @AlekSi , can you help provide context to repro this issue.
we want to repro this using mongodump/mongorestore/mongoimport/mongoexport, i've tried with doc with timestamp(0,0), but can't repro it.

I assume it shows this failure if there's specific timestamp field in doc or using some parameter(i.g. --archive or --oplog).
can you share the sample data and command?

[AlekSi]

It is not data-specific. Just use the latest mongoimport version, which sets this field in the insert commands.

[duanpei-ms]

Hi @AlekSi @kbaegis, you mean that this issue will occur as long as mongoimport, regardless of the data?
I have tried using 100.12.0 to reproduce it, and it works.

Could you give me an example? Or if my understanding is incorrect, please feel free to point it out. Thanks!

[chilagrow]

I see two points to address here. First, inserting zero timestamp Timestamp(0,0) has discrepancy. Second, bypassEmptyTsReplacement isn't supported.

Here's the version information of the PostgreSQL and documentdb

postgres=# SELECT documentdb_api.binary_extended_version();
            binary_extended_version             
------------------------------------------------
 0.104.0 gitref: ferretdb sha:560b14b buildId:0
(1 row)

postgres=# SELECT version();
                                                       version                                                       
---------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.5 (Debian 17.5-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
(1 row)

postgres=# SET documentdb_core.bsonUseEJson TO true;
SET

The first point about the discrepancy, inserting zero value timestamp Timestamp(0,0) to MongoDB actually inserts Timestamp({ t: 1748431087, i: 1 }) which is the current timestamp.

MongoDB 8.0.6:

test> db.runCommand({insert: "ts", documents: [{_id : "zero", v: Timestamp(0,0)}]})
{ n: 1, ok: 1 }
test> db.ts.find()
[ { _id: 'zero', v: Timestamp({ t: 1748431087, i: 1 }) } ]

In documentdb, inserting { "$timestamp" : { "t" : 0, "i" : 0 } } does not convert to the current timestamp. I believe it should behave the same.

postgres=# SELECT * FROM documentdb_api.insert('test', '{"insert": "ts", "documents": [{"_id" : "zero", "v": {"$timestamp": {"t": 0, "i": 0}}}]}', NULL);
NOTICE:  creating collection
                               p_result                               | p_success 
----------------------------------------------------------------------+-----------
 { "n" : { "$numberInt" : "1" }, "ok" : { "$numberDouble" : "1.0" } } | t
(1 row)

postgres=# SELECT cursorpage FROM documentdb_api.find_cursor_first_page('test', '{"find": "ts"}', 0);
                                                                                          cursorpage                                                                                           
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 { "cursor" : { "id" : { "$numberLong" : "0" }, "ns" : "test.ts", "firstBatch" : [ { "_id" : "zero", "v" : { "$timestamp" : { "t" : 0, "i" : 0 } } } ] }, "ok" : { "$numberDouble" : "1.0" } }
(1 row)

The second point about bypassEmptyTsReplacement support, inserting Timestamp({ t: 0, i: 0 }) to MongoDB inserts zero timestamp without converting to the current timestamp, the inserted value is Timestamp({ t: 0, i: 0 }).

MongoDB 8.0.6:

test> db.runCommand({insert: "ts", bypassEmptyTsReplacement: true, documents: [{_id : "zero_bypass_empty", v: Timestamp(0,0)}]})
{ n: 1, ok: 1 }
test> db.ts.find()
[
  { _id: 'zero', v: Timestamp({ t: 1748430181, i: 1 }) },
  { _id: 'zero_bypass_empty', v: Timestamp({ t: 0, i: 0 }) }
]

The documentdb currently doesn't support bypassEmptyTsReplacement and returns error.

postgres=# SELECT * FROM documentdb_api.insert('test', '{"insert": "ts", "bypassEmptyTsReplacement": true, "documents": [{"_id" : "zero", "v": {"$timestamp": {"t": 0, "i": 0}}}]}', NULL);
ERROR:  BSON field 'insert.bypassEmptyTsReplacement' is an unknown field

Inserting Timestamp(0,0) should convert to current time such as Timestamp({ t: 1748431087, i: 1 }). Only if bypassEmptyTsReplacement option is set, it should preserve Timestamp(0,0). As far as I'm aware bypassEmptyTsReplacement option is used in tools such as mongoimport (version 100.12.1).