Question regarding update vs insert

[mkhorton]

The MongoStore does not provide an insert option, only an update option. This does a bulk_write but as a list of ReplaceOne due to how maggma deals with key handling, which presumably(?) is a lot slower than just a list of insert ops.

I'm building a large number of documents but I've dropped down to using store.collection.insert_many() since the update seems prohibitive.

Is the lack of insert intentional? Has this been problematic for others?

[shyamd]

I've found bulk unordered ReplaceOne with upsert=True is almost as performant as insert_many. The only time this is an issue is when there isn't an index on the key field which can happen, and then its much slower. I've been hesitant to enforce indexes in the store themselves but considering making it a default to call ensure_index on the key and last_updated fields in the connect method.

[mkhorton]

I've been hesitant to enforce indexes in the store themselves but considering making it a default to call ensure_index on the key and last_updated fields in the connect method.

Ah, I thought this was the case -- in my case, this was building to a new collection (i.e. without indices) and thought that indices on the key would be generated.

Given the performance of maggma determines critically on indices being available for both the key field and last_updated field, I would definitely be on board with ensuring indexes by default.