Question on duplication

[Crispy1975]

Just started testing with monstache and so far very impressed. I have a quick question regarding duplicates; specifically how are MongoDB _id properties mapped to ElasticSearch, and how when a document in MongoDB is updated is the correct ElasticSearch document updated?

[rwynn]

The default mapping is as follows:

mongodb database -> elasticsearch index
mongodb collection -> elasticsearch type
mongodb document _id -> elasticsearch document id

Any operation on a mongo document (insert, update, delete) will be recorded in the oplog with it's database, collection, and _id and that individual operation will eventually be folded into a bulk index command to ES (with other operations) where the index, type and _id are generated according to the mapping above. That is how it manages to index, update, or delete the correct ES document.

However, I think you may be correct about the potential for collisions since mongo _ids are unique at the collection level (not guaranteed across the database) while ES ids must be unique at the index level.

According to link, duplicate _ids across the database are unlikely with autogenerated _ids.

Regardless, I should change the default mapping such that

elasticsearch index <- mongodb database . mongodb collection

Thanks for giving monstache a try and reporting this issue.

[Crispy1975]

Awesome, thanks. 👍