Skip to content

Commit

Permalink
Consolidate summation in _ravel_shape_indices (#72)
Browse files Browse the repository at this point in the history
Instead of using the Python `sum` function on Dask Arrays, which will
handle the reduction one pair at a time thus generating a large graph,
use `atop` with a special summation kernel. This handles summation of
all of the Dask Arrays in one function and returns the result. Thus
simplifying the addition into one node. To simplify the Dask graph
further, move the creation of singleton axes into the
`_ravel_shape_indices` kernel function as well. This avoids adding these
steps needlessly to the graph of each `arange`. Plus NumPy should be
able to do these for basically free. As a result graph construction by
`_ravel_shape_indices` should be merely using `arange` to construct
indices for each dimensions and combining these into the resultant
raveled indices with `atop` and the custom kernel function.
  • Loading branch information
jakirkham committed Sep 30, 2018
1 parent 72d2c69 commit 783c109
Showing 1 changed file with 17 additions and 3 deletions.
20 changes: 17 additions & 3 deletions dask_image/ndmeasure/_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,21 +50,35 @@ def _get_label_matches(labels, index):
return lbl_mtch


def _ravel_shape_indices_kernel(*args):
args2 = tuple(
a[i * (None,) + (slice(None),) + (len(args) - i - 1) * (None,)]
for i, a in enumerate(args)
)
return sum(args2)


def _ravel_shape_indices(dimensions, dtype=int, chunks=None):
"""
Gets the raveled indices shaped like input.
"""

indices = sum([
indices = [
dask.array.arange(
0,
numpy.prod(dimensions[i:], dtype=dtype),
numpy.prod(dimensions[i + 1:], dtype=dtype),
dtype=dtype,
chunks=c
)[i * (None,) + (slice(None),) + (len(dimensions) - i - 1) * (None,)]
)
for i, c in enumerate(chunks)
])
]

indices = dask.array.atop(
_ravel_shape_indices_kernel, tuple(range(len(indices))),
*sum([(a, (i,)) for i, a in enumerate(indices)], tuple()),
dtype=dtype
)

return indices

Expand Down

0 comments on commit 783c109

Please sign in to comment.