Decode "timestamp" from oplog.timestamp

[howardkhl]

On the wiki it says:
The exact format of this file depends on MongoDB's toplogy. For a single replica set, the format is:

["oplog name", timestamp]

An example would be:

["Collection(Database(MongoClient('localhost', 27017), u'local'), u'oplog.rs')", 6185588281674039941]

What format is this timestamp in, and how do you convert say, 6185588281674039941, to a normal date?

Thanks.

[howardkhl]

In local.oplog.rs, oplog entry has timestamp value like this:

"ts" : Timestamp(1440194492, 391)

How is it relate to the timestamp format in oplog.timestamp? Perhaps this in oplog_manager.py?

timestamp = util.bson_ts_to_long(entry['ts'])

If so, is there a way to convert this format back to a human readable date in mongo shell?

[howardkhl]

Okay, found a solution for this.

util.py has a method long_to_bson_ts, use Python to run the method directly:

python -c 'import util; print util.long_to_bson_ts(<val>)'

For example:

python -c 'import util; print util.long_to_bson_ts(6185145118358503425)'

And you will get a Timestamp object back.

Timestamp(1440091319, 1)

In python, use datetime to convert Timestamp to human readable form:

python
>>> from datetime import datetime
>>> str(datetime.fromtimestamp(1440091319))

It will return:

'2015-08-20 10:21:59'

For comparison purpose, this command will return current system time:

>>> str(datetime.now())

[behackett]

Please note that the BSON Timestamp type is seconds since the Unix epoch plus an increment to represent multiple operations within the same second. See this for more information:

http://docs.mongodb.org/master/reference/bson-types/#timestamps

[upenblog]

Thanks@ behackett

followed the same steps but I get current time stamp no matter what value you I use
rs0:SECONDARY> new ISODate(new Timestamp(1442245369, 356) * 1000)
ISODate("2015-09-15T20:28:11.099Z")

rs0:SECONDARY> new ISODate(new Timestamp(1442244777, 386) * 1000)
ISODate("2015-09-15T20:28:30.838Z")

[howardkhl]

@upenblog Please follow updated step. Go to python and use datetime.

[howardkhl]

troubleshooting notes, if you get:

ImportError: No module named util
Make sure you git pull mongo-connector project, and you run the python command from mongo-connector/mongo_connector directory where util.py is.

ImportError: No module named mongo_connector.compat
Edit util.py, change line 26: from mongo_connector.compat import reraise to from compat import reraise

[howardkhl]

Extra credit: if you need to convert from bson timestamp to long (integer format used in oplog.timestamp):
python -c 'import util; from bson.timestamp import Timestamp; print util.bson_ts_to_long(Timestamp(<val>, <inc>))'

E.g.
python -c 'import util; from bson.timestamp import Timestamp; print util.bson_ts_to_long(Timestamp(1490951863, 8))'
6403589491495272456

[leroichou]

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "util.py", line 22, in <module>
from bson import Timestamp
ImportError: cannot import name Timestamp

It failed when I imported the util module. How do I resolve the problem like this?

All right, I know what question is. I need to install the pymongo module.
#pip install pymonngo

[howardkhl]

Update, I've tried the following on my mac terminal and no dependencies issue. Convert from Epoch time to bson format (oplog) and vice versa.

https://atharva-inamdar.medium.com/understanding-mongodb-oplog-249f3996f528

1. Install pymongo
   pip install pymongo
2. In Python3 terminal
   python3
3. Copy paste:

from bson import Timestamp
from datetime import datetime
def ts_to_int(ts):
  return (ts.time << 32) + ts.inc
def int_to_ts(t):
  return Timestamp(t>>32, t & (2**32)-1)

4. Try it out:

>>> ts=Timestamp(datetime.utcnow(), 1)
>>> print(ts_to_int(ts))
7052809609722986497
>>> ts=Timestamp(1642100404, 1)
>>> print(ts_to_int(ts))
7052767531928387585