Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Modify mean chunk functions to return dicts rather than arrays #4513
These functions return two array chunks each:
We need to return these as a single intermediate value.
Previously we did this by constructing an empty numpy array and then
Now we return a dict containing two arrays. This is nice in a few ways,
This is also somewhat suboptimal in a distributed setting because we no longer get special-cased handling of numpy arrays like we used to. The scheduler will probably miscount the number of bytes that these objects hold and will probably not use ideal serialization and compression. I think that this is probably ok because they're likely to be small, but it's worth pointing out.
After this we would like to implement the same fix for other functions,
referenced this pull request
Feb 20, 2019
If you like I'm also happy to hand this entire commit to you to finish up if you're interested :)…
On Thu, Feb 21, 2019 at 7:05 AM Peter Andreas Entschev < ***@***.***> wrote: By the way, @mrocklin <https://github.com/mrocklin> if you want to get your commit fixed and merge it, feel free to leave moment_* to me, I just found out it's the next one in line for Dask-GLM+CuPY. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4513 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszL8B-HVZPYIAEILdEMsnVJ3dt9Kiks5vPrXPgaJpZM4bEV_j> .
Thinking about extending this to moment, it looks like there is a little bit of a challenge here:
empty = empty_lookup.dispatch(type(n)) M = empty(n.shape + (order - 1,), dtype=dtype) for o in range(2, order + 1): M[..., o - 2] = _moment_helper(Ms, ns, inner_term, o, sum, kwargs)
Here we construct an empty array and then assign into it.
Instead, we might construct many arrays and then concatenate them together.
xs = [_moment_helper(Ms, ns, inner_term, o, sum, kwargs) for o in range(2, order + 1)] M = np.stack(x2, axis=-1)
(I might have the axis= keyword above wrong. Also depending on the shape of the output of
@mrocklin that might work, but couldn't that cause a potential performance issue? I guess the stacked blocks will all remain different memory blocks, right? My concern is more on the case if stacking doesn't cause all the blocks to be moved to a single memory block, thus involving allocation and copy.
I suspect that the data at this stage of the computation will probably be pretty small (it has already gone through a reduction along at least one axis), so I'm not too concerned about performance.
I think that we can avoid this. We no longer construct an empty array and then assign into it. Instead we create many arrays and then call
referenced this pull request
Feb 22, 2019
I just tried this new test you added and it fails also in current master (i.e., before the changes in this PR). I think this is something we could (maybe even should) tackle in a different issue?