You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now, we're dumping the whole _BatchManager in a JSON file. This is fine for pipelines that doesn't generate too much data, but doesn't work well for pipeline generating too much data (embeddings for example).
We need to dump the data of each _BatchManagerStep in a different file (or files), so when loading the _BatchManager back we don't have issues.
The text was updated successfully, but these errors were encountered:
Right now, we're dumping the whole
_BatchManager
in a JSON file. This is fine for pipelines that doesn't generate too much data, but doesn't work well for pipeline generating too much data (embeddings for example).We need to dump the
data
of each_BatchManagerStep
in a different file (or files), so when loading the_BatchManager
back we don't have issues.The text was updated successfully, but these errors were encountered: