-
Notifications
You must be signed in to change notification settings - Fork 273
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The datasets conditioned/gsim_XXX
are using too much disk space
#9387
Comments
Actually the only solution is to reduce the number of sites, since the memory/disk space occupation is quadratic with the number of sites. |
Why doesn't storing them in the .tmp.hdf5 work? This is data needed only during the calculation and doesn't need to be stored in the final calc.hdf5 |
Because you will soon run out of disk space, this is how Cata discovered the issue. Also, once you start storing 100+GB then reading the data will kill your calculation (out of memory or so slow to be impossible to run). No matter how big is your machine, a quadratic calculation will run out of resources pretty soon. You would need an algorithm not quadratic with the number of sites. |
Opening this issue again as it still persists. The issue is not related to having too many sites in the calculation. It was that the |
You are partially right @raoanirudh , but my point still stand that calculations with too many points will be impossible. The only solution I see for Aristotle calculations is to use a large enough |
As reported by @CatalinaYepes . A solution could be to store them in the .tmp.hdf5 file. Otherwise, we could revert #9094.
The text was updated successfully, but these errors were encountered: