You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For many use cases, running intake top of a directory of mspgack or jsonl files is not suitable. The databroker-unpack utility should provide an option to copy Documents from the files into MongoDB and auto-generate a suitable catalog file.
I didn't do this in the first pass because it was not obvious to me how to spell, or how we might want to extend it. Initial thought:
databroker-unpack inplace DIRECTORY NEW_CATALOG_NAME # current functionality
Now, databroker-pack is intentionally "opinionated" and only supports a hard-coded subset of suitcases (msgpack, jsonl). Should we do the same for -unpack? I think so, because it enables us to auto-generate a working catalog file.
The text was updated successfully, but these errors were encountered:
Maybe a --merge flag that will try to insert into an existing catalog (rather than creating a new one). You can imagine having behind the scenes logic to sort out if we can side-load (ex msgpack -> msgpack) by just copy files, but fallback to db_new.insert(...).
We probably need a set of helpers for creating new catalogs (of what ever backing) and dumping the config files in the correct place.
For many use cases, running intake top of a directory of
mspgack
orjsonl
files is not suitable. Thedatabroker-unpack
utility should provide an option to copy Documents from the files into MongoDB and auto-generate a suitable catalog file.I didn't do this in the first pass because it was not obvious to me how to spell, or how we might want to extend it. Initial thought:
and
Now,
databroker-pack
is intentionally "opinionated" and only supports a hard-coded subset of suitcases (msgpack, jsonl). Should we do the same for-unpack
? I think so, because it enables us to auto-generate a working catalog file.The text was updated successfully, but these errors were encountered: