MongoDB Output #159
MongoDB Output #159
Conversation
There was a problem hiding this comment.
Looks good Dan, thanks for adding this! I forgot why the old MongoDB store was removed, but maybe I was too lazy to port it to the new frontend/backend paradigm... This also fixes many issues with that old code, such as support for multi-row chunks and array fields.
strax/storage/mongo.py
Outdated
| limit is respected! | ||
| """ | ||
|
|
||
| try: |
There was a problem hiding this comment.
pymongo is now a dependency (due to using json_util also in the files backend), so this check is not needed.
strax/storage/mongo.py
Outdated
| self.log.debug(f"{key} is NOT in cache.") | ||
| raise strax.DataNotAvailable | ||
|
|
||
| def _read_meta(self, coll_name): |
There was a problem hiding this comment.
Thanks for the suggestions. Updated.
One thing I just noticed is that I never call 'drop'. Is it standard in strax to leave partial data written on disk or clear it in case of a crash? I saw the allow_partial option so I'm assuming I could use that if I want to handle it somehow.
There was a problem hiding this comment.
Thanks! Indeed it's standard to leave it; this dates back to when we thought online deletion was necessary. The only time strax clears data is while overwriting, I guess you could drop the collection if it already exists when you init the saver, just like in https://github.com/AxFoundation/strax/blob/master/strax/storage/files.py#L277. The frontend has an allow_overwrite option that can prevent savers getting created if the data already exists and isn't corrupted (unless you say you want to overwrite).
Started with a mongodb store that Jelle wrote then removed from strax ~1 year ago and updated it to the frontend/backend/saver paradigm. It's for use in the online monitoring where some basic plugins will dump info to MongoDB for display online.