Join GitHub today
[WIP] Adds data stores and references #37
This is still a WIP, more testing and internal API contracts need to be evaluated
In the glue api there is now a
Internally, In place of just specifying an encoder, there's now an encoder_name and a store_name ("json", "notebook") associated with data managers. Data managers can implement encoding or storing, or both capabilities. One can implement ("text", None) and (None, "s3") independently. This lets simple encoders return string or bytes that the stores can save while allowing for more complex store and recall mechanism to encode and store at the same time (e.g. dataframe to multi-file s3 parquet via ("arrow", "s3")).
To facilitate the contract changes there is now a scrapbook v2 schema. Loading v1 or v2 data is transparent to the library user.