-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Means of quadratic quantities #2
Comments
Cubed completed this workload using only 1.5GB of RAM! https://gist.github.com/TomNicholas/8366c917349b647d87860a20a257a3fb |
I would like to try this problem with cubed using real data instead of random data. @dcherian (/anyone) if you know, can you explain a little more about the context of this issue please? So that I understand if/how I might be able to use some publicly available zarr data to create a representative benchmark case that includes I/O. Something about anomalies of GCM data... 😅 |
cc @robin-cls who opened the original xarray issue |
FYI I could track this problem down to the way dask performs the topological sort / prioritization of tasks, see dask/dask#10384 This example should work trivially when either is true:
|
Anecdotally I think the performance is much better when you only compute one array, yes. |
Just a heads up. I'm working for a fix for this in dask/dask, see dask/dask#10535 Preliminary results look very promising This graph show the memory usage for a couple of runs with increasing size in the time partition. This increases basically number of tasks but keeps the individual chunks and the algorithm constant.
I'm also looking at other workloads. If you are aware of other stuff that should be constant or near-constant in memory usage but isn't, please let me know! |
From pydata/xarray#6709
This example calculates
ds.u.mean()
,ds.v.mean()
, and(ds.u * ds.v).mean()
all at the same timeWith dask, we get not-so-great memory use. (Colors are for different values of "worker saturation")
The text was updated successfully, but these errors were encountered: