Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve saving _BatchManager content for cache #487

Closed
gabrielmbmb opened this issue Mar 27, 2024 · 0 comments
Closed

Improve saving _BatchManager content for cache #487

gabrielmbmb opened this issue Mar 27, 2024 · 0 comments
Assignees
Milestone

Comments

@gabrielmbmb
Copy link
Member

Right now, we're dumping the whole _BatchManager in a JSON file. This is fine for pipelines that doesn't generate too much data, but doesn't work well for pipeline generating too much data (embeddings for example).

We need to dump the data of each _BatchManagerStep in a different file (or files), so when loading the _BatchManager back we don't have issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

No branches or pull requests

3 participants