Add atop(concatenate=False) keyword argument #1609
Conversation
|
cc @shoyer |
| function should expect an iterator of blocks on any array that holds that | ||
| index. | ||
| function should expect an iterable of blocks on any array that holds that | ||
| index. To receieve an array concatenated along the dummy dimension pass |
There was a problem hiding this comment.
Instead:
To receive arrays concatenated along contracted dimensions instead pass concatenate=True
|
Yes, I found the nested lists of arrays passed previously quite unintuitive, and also a little misleading, because it suggested that the arrays could be processed in a streaming fashion (when in reality they're loaded into memory all at once). |
This allows atop to pre-concatenate arrays before sending to the user defined function.
mrocklin
force-pushed
the
atop-concatenate
branch
from
c6b0288
to
1d7a855
Compare
Historically we used to use iterators and stream things through. This is no longer the case though. |
|
Merging. |
OK, that could explain part of my confusion... |
This allows atop to pre-concatenate arrays before sending to the user
defined function.
Previously we would send an iterable of chunks to the user-defined-function. This allows us to optionally pre-concatenate those arrays ahead of time, simplifying the effort on the user's side.