error: 'utf8' codec can't decode byte 0xa1

[islue]

$ grep TailThread-2 /var/log/mongodb-consistent-backup/backup.stats.20170525_0557.log
...
[2017-05-25 07:09:30,587] [INFO] [TailThread-2] [TailThread:status:60] Oplog tailer rs0/ip-10-1-16-127:27017 status: 1227990 oplog changes, ts: Timestamp(1495696169, 6)
[2017-05-25 07:10:01,011] [INFO] [TailThread-2] [TailThread:status:60] Oplog tailer rs0/ip-10-1-16-127:27017 status: 1233520 oplog changes, ts: Timestamp(1495696200, 2)
[2017-05-25 07:10:22,763] [CRITICAL] [TailThread-2] [TailThread:run:102] Tailer rs0/ip-10-1-16-127:27017 error: 'utf8' codec can't decode byte 0xa1 in position 0: invalid start byte

Sorry, 'utf8' codec issue again.
I have checked the codes around but can't find something calling decode. Any idea?

[islue]

76:            cursor = db.oplog.rs.find(query, cursor_type=CursorType.TAILABLE_AWAIT, oplog_replay=True)
...
81:                        doc = cursor.next()

I guess here comes the problem. I suppose that we can add an option unicode_decode_error_handler="ignore" for MongoClient. Is it safe for oplog data?

[timvaillancourt]

Yes, utf8 correctness does not need to be checked on oplog items. This follows the common theme in MongoDB that the server (and thus oplog.rs) is the source of correctness, which is an assumption followed by MongoDB's own replication and all tools (mongodump, mongorestore, etc).

Pymongo is defaulting to being more strict here and it can be relaxed for any oplog items only. That said, this does mean that you have fields in documents that do not adhere to MongoDB standards, possibly dots "." in field names, etc. You may want to check on this in case those standards become enforced in future versions of MongoDB, currently they're not.