Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How does bibat perform with large datasets? #82

Open
teddygroves opened this issue Apr 5, 2024 · 0 comments
Open

How does bibat perform with large datasets? #82

teddygroves opened this issue Apr 5, 2024 · 0 comments

Comments

@teddygroves
Copy link
Owner

See reviewer comment here.

I can think of a few relevant topics:

  • some arviz storage formats are faster than others. Currently bibat allows either zarr or json: these should be compared with each other and the other options.
  • there is no provision for running inferences in parallel. This could potentially save a lot of time for complex projects with large datasets and wouldn't be too hard to do.
  • there is no option to not load all data in memory at once. Implementing a 'big data' Bayesian workflow where chunks of data are processed separately would be pretty tricky, but it would be interesting to see at exactly what scale memory problems start and then tell users about this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant